COS Migration is an all-in-one tool that integrates the COS data migration feature. You can use it to migrate local data to COS through simple configurations and steps. It has the following features:
- COS Migration only supports UTF-8 encoding.
- If you use this tool to upload a file that already has the same name, the existing file will be overwritten. You need to configure the tool to skip files with the same name.
- Use the migration service platform preferably for scenarios other than local data migration.
Windows, Linux, and macOS.
Download COS Migration here.
Decompress the package and save it to a directory, for example:
Decompress the package and save it to a directory, for example:
unzip cos_migrate_tool_v5-master.zip && cd cos_migrate_tool_v5-master
The structure of the properly decompressed COS Migration tool is as follows:
COS_Migrate_tool |——conf #Directory of the configuration file | |——config.ini #Migration configuration file |——db #Store the record of successful migrations |——dep #JAR package generated by compilation of the main logic of the program |——log #Log generated during tool execution |——opbin #Script for compilation |——src #Source code of the tool |——tmp #Temporary file storage directory |——pom.xml #Project configuration file |——README #Readme document |——start_migrate.sh #Migration start script for Linux |——start_migrate.bat #Migration start script for Windows
dbdirectory mainly records the IDs of files successfully migrated by the tool. Each migration job will first compare the records in the
dbdirectory. If the ID of the current file has already been recorded, the current file will be skipped; otherwise, it will be migrated.
logdirectory keeps all the logs generated during tool migration. If an error occurs during migration, check
error.login this directory first.
Before running the migration start script, modify the
config.ini file (path:
./conf/config.ini) first. This file contains the following parts:
type indicates the migration type, which you need to enter based on your migration needs. For example, to migrate local data to COS, configure
Currently, the following migration types are supported:
|migrateLocal||Migration from local system to COS.|
You can configure a migration job based on your actual needs, including its destination COS bucket information.
# The common configuration section of the migration tool includes account information to be migrated to the destination COS bucket. [common] secretId=COS_SECRETID secretKey=COS_SECRETKEY bucketName=examplebucket-1250000000 region=ap-guangzhou storageClass=Standard cosPath=/ https=off tmpFolder=./tmp smallFileThreshold=5242880 smallFileExecutorNum=64 bigFileExecutorNum=8 entireFileMd5Attached=on daemonMode=off daemonModeInterVal=60 executeTimeWindow=00:00,24:00 outputFinishedFileFolder=./result resume=false skipSamePath=false
|bucketName||Name of the destination bucket in the format of
|region||Region information of the destination bucket. For the abbreviations for COS regions, see Regions and Access Endpoints.||-|
|storageClass||Storage class for the migrated data. Valid values:
|cosPath||COS path to migrate to.
|https||Whether to transfer over HTTPS. Valid values: on: Yes; off: No. It takes time to enable transfer over HTTPS, which is suitable for scenarios that demand high security.||off|
|tmpFolder||The directory used to store temporary files when data is migrated from another cloud storage service to COS, which will be deleted after the migration is completed. The format must be an absolute path:
Linux uses a slash (/) as the delimiter, for example,
Windows uses two backlashes (\) as the delimiter, for example,
The default value is the
|smallFileThreshold||Number of bytes as the threshold for small files. If the size is greater than or equal to this threshold, multipart upload will be used; otherwise, simple upload will be used. Default value: 5 MB||5242880|
|smallFileExecutorNum||Concurrency for uploading small files (smaller than
|bigFileExecutorNum||Concurrency for uploading large files (greater than or equal to
|entireFileMd5Attached||The migration tool calculates the MD5 of the entire file and stores it in the
|daemonMode||Whether to enable daemon mode. Valid values: on: Yes; off: No. In daemon mode, the program will keep performing synchronization. The synchronization interval is configured by the
|daemonModeInterVal||Time interval in seconds between two rounds of synchronization.||60|
|executeTimeWindow||Execution time window with a granularity in minute, which defines the time period when the migration tool runs jobs. For example:
|outputFinishedFileFolder||This directory stores the results of successful migration jobs, and the result files are named by date, for example,
|resume||Whether to continue with the result of the last run and traverse through the list of files from the source. The tool starts from scratch by default.||false|
|skipSamePath||Whether to skip the existing file if a file with the same name already exists in COS. By default, the tool does not skip the existing file; that is, it overwrites the existing file.||false|
Configure each section according to the migration type described in
[migrateType]. For example, if
[migrateType] is configured as
type=migrateLocal, you will only need to configure the
3.3.1 Configure a local data source migrateLocal
To migrate from a local system to COS, configure this section. The specific configuration items and descriptions are as follows:
# Configuration for migration from a local system to COS [migrateLocal] localPath=E:\\code\\java\\workspace\\cos_migrate_tool\\test_data excludes= ignoreModifiedTimeLessThanSeconds=
|localPath||Absolute path of the local directory
|excludes||Absolute path of the directory or file to be excluded, indicating some directories or files under
|ignoreModifiedTimeLessThanSeconds||Exclude files that have an update time less than a certain period of time from the current time (in seconds). This item is left blank by default, indicating files are not to be filtered by the time specified by
Double-click start_migrate.bat to run the tool.
Run the following command to read the configuration from the
Run the following command to read the configuration from command lines for some parameters:
sh start_migrate.sh -Dcommon.cosPath=/savepoint0403_10/
- The tool supports reading configuration items in two ways: command line or configuration file.
- The command line takes priority over the configuration file, i.e., for the same configuration item, parameters in command lines take priority.
- Reading configuration items from command lines allows you to run different migration jobs at the same time, provided that key configuration items (such as bucket name, COS path, and source path to be migrated) in the two jobs are not exactly the same. Concurrent migration can be achieved because different migration jobs are written into different
dbdirectories. Refer to
dbinformation in the tool structure above.
- Configuration items are in the format of
sectionNameis the section name of the configuration file.
sectionKeyis the name of the configuration item in the section.
sectionValueis the value of the configuration item in the section. The COS path to which data is to be migrated should be in the format of
COS Migration has a status. Successful migrations will be recorded in the format of
KV in the
leveldb file under the
db directory. Before each migration, check whether the path to which data is migrated has been recorded in the
db directory. If yes, and its attribute is the same as that in
db, the migration will be skipped; otherwise, the migration will be executed. The attribute varies by migration type. For local migration,
mtime determines whether to migrate. For migration from other cloud storage services and bucket replication, the etag and length of the source file determine whether to migrate. Therefore, we search for records of successful migrations in the
db directory rather than in COS. If a file is deleted or modified via COSCMD or the console rather than the migration tool, the migration tool cannot detect this change, and the file will not be re-migrated.
dbdirectory according to the specified migration type to determine whether upload is allowed.
inprogressindicates migration is in progress,
okindicates succeeded, and
condition_not_matchindicates file fails to meet migration conditions (such as
excludes) and is skipped. Details about the failure can be viewed in the error log. The execution process is as shown below:
If an exception such as migration failure or execution error occurs when you use the COS Migration, troubleshoot as instructed in COS Migration.