The traditional rsync tool does not support specifying the start/end time for data migration/backup. As a result, the migration might be performed during business peak hours, consuming the computing, network, storage, and other resources. To solve this problem, CFS introduces the Filetruck tool for users to control the start/end time of data migration tasks.
The following describes how to use Filetruck to migrate/back up data.
Before you start, find the image of CFS Filetruck in the Tencent Cloud Market.
Filetruck starts a migration task based on the configuration file in
ini format. The configuration file of a new task needs to be created by the user manually and run as the admin. The most basic configuration file has the following content:
# Absolute path of the source directory SourceDirPath=/path/to/sourceDir # Absolute path of the destination directory TargetDirPath=/path/to/targetDir
You can copy the code above to a
newTask.ini file and modify the source/destination directories as needed. Then, you can start a new task by running the
filetruck_client -c newTask.ini command as the root account.
If the new task is started successfully, Filetruck will return the following message that includes the
# filetruck_client -c newTask.ini TaskId=1
If the configuration is incorrect, the new task might fail to be started. In this case, Filetruck will return the corresponding error message for you to modify the configuration file:
# filetruck_client -c newTask.ini Error: Directory does not exist: /path/to/sourceDir
In most cases, you only need to specify
TargetDirPath to create a migration task. If you need to set the start/end time (
EndDate), bandwidth limit (
BandwidthLimitInKbps), and the destination file matching rules (
ExcludeRule), you can use the Filetruck advanced task configuration file.
An advanced configuration file has the following content:
# Absolute path of the source directory (required) SourceDirPath=/path/to/sourceDir # Absolute path of the destination directory (required) TargetDirPath=/path/to/targetDir # Task start time, formatted as YYYY MMM DD hh:mm:ss (optional) StartDate=2020 Aug 02 11:22:33 # Task end time, formatted as YYYY MMM DD hh:mm:ss (optional) EndDate=2020 Aug 02 12:22:33 # Bandwidth limit (in KiB/s) in the range [0, 2147483647], where 0 indicates no limit (optional) BandwidthLimitInKbps=1024 # Matching rule for files to migrate (optional) IncludeRule=*.jpg # Matching rule for files to skip (optional) ExcludeRule=*.png
TargetDirPathare required and should be set to the absolute paths of the directories.
ExcludeRuleare optional and do not need to be specified in the configuration file.
EndDatemust be in the
yyyy mmm dd HH:MM:SSformat. For example, to specify 20:20:08 on August 8, 2020, set the parameter to
2020 Aug 08 20:20:08instead of
2020 Aug 8 20:20:8.
BandwidthLimitInKbpsis [0, 2147483647], where
0indicates unlimited bandwidth. Note: The Filetruck migration speed is determined by the CPU/memory/bandwidth configurations of the host it is installed on, the network condition of the source/destination paths, and the size of the file to migrate/back up. In most cases, if the host configuration and egress bandwidth are high, and the file size is large, the migration/backup speed will be faster. For example, if the host is configured with 8-core CPU, 16 GB memory, and 1.5 Gbps bandwidth, and the size of the local file to migrate to the CFS high-performance file system is 4 KB, the speed would be about 40 KB/s; if the file size is 1 TB, the speed would be about 140 MB/s.
ExcludeRuleare as follows:
IncludeRuletakes effect only when
ExcludeRuleare specified, files in the source directory would be considered as a whole. Files to migrate would be those excluded by
ExcludeRuleas well as those included by both
*matches all multi-characters except
?matches single-characters except
You can run the
filetruck_client -l command to query all historical tasks (including ongoing and completed tasks) of Filetruck as follows:
# filetruck_client -l TaskId State FileCount SentBytes Speed Progress 1 Partial Done 59 0 0B/s 3% 2 All Done 1 1073872935 511K/s 100% 3 All Done 2 1073872935 511K/s 100% 4 Partial Done 1 0 0B/s 3% 5 All Done 2 1073872935 511K/s 100% 6 All Done 2 1073872935 511K/s 100% 7 All Done 1285 138796240 502K/s 100% 8 All Done 1 0 0B/s 100% 9 All Done 0 0 0B/s 100% 10 Running 712 82799297 449K/s 24%
All Done: all completed
Partial Done: completed partially (stopped due to timeout or user cancellation)
Progressis not calculated yet,
-will be displayed.
inodevalue of the file will be changed.
You can run the
filetruck_client -t TASK_ID with
TASK_ID specified to query the detailed information of a task:
# filetruck_client -t 3 TaskId: 3 TaskState: All Done SourceDirPath: /mnt/cfs0/ TargetDirPath: /mnt/cfs1/ BeginDate: 2020 Dec 02 15:13:20 EndDate: 2020 Dec 02 15:30:25 SentFilesCount: 2 SentDataSizeInByte: 1073872935 SentSpeed: 511K/s TaskProgress: 100%
If the value of
TASK_ID is invalid, the following error message will be displayed:
# filetruck_client -t 999 Error: Task with ID=999 is not founded.
You can run the
filetruck_client -k TASK_ID command with
TASK_ID specified to cancel an ongoing task:
# filetruck_client -k 10 Success
If an ongoing migration task is canceled, the destination directory of the unfinished file will be released. This is to avoid leaving incomplete files in the destination directory.
You can run the
filetruck_client -h command to obtain the help information of Filetruck as follows:
CFS-Filetruck version: 0.1.3b Copyright (C) 2020 Tencent Inc. All rights reserved. Link: https://intl.cloud.tencent.com/products/cfs CFS-Filetruck makes file migration easier. Usage: Create a new task: filetruck_client -c /path/to/new/task/config/file.ini Get task state by task id: filetruck_client -t TASK_ID Cancel/Kill task by task id: filetruck_client -k TASK_ID List all tasks: filetruck_client -l Print help: filetruck_client -h