- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Console Guide
- Console Overview
- Bucket Management
- Bucket Overview
- Creating a Bucket
- Deleting a Bucket
- Querying Buckets
- Emptying a Bucket
- Setting Access Permission
- Setting Bucket Encryption
- Setting Hotlink Protection
- Setting Origin-Pull
- Setting CORS
- Setting Versioning
- Setting Up a Static Website
- Setting Lifecycle
- Setting Logging
- Accessing Bucket List Using Sub-Account
- Adding a Bucket Policy
- Setting Log Analysis
- Setting INTELLIGENT TIERING
- Setting Inventory
- Domain Name Management
- Setting Bucket Tags
- Setting Cross-Bucket Replication
- Enabling Global Acceleration
- Object Management
- Uploading Objects
- Downloading Objects
- Copying Objects
- Viewing Object Information
- Searching for Objects
- Sorting and Filtering Objects
- Direct Upload to ARCHIVE
- Modifying Storage Classes
- Deleting Incomplete Multipart Uploads
- Setting Object Access Permission
- Setting Object Encryption
- Custom Headers
- Deleting Objects
- Restoring Archived Objects
- Folder Management
- Data Extraction
- Setting Object Tags
- Exporting Object URLs
- Restoring a Historical Object Version
- Batch Operation
- Monitoring Reports
- Data Processing
- Data Processing Workflow
- Application Integration
- User Tools
- Tool Overview
- Installation and Configuration of Environment
- COSBrowser
- COSCLI (Beta)
- COSCLI Overview
- Download and Installation Configuration
- Common Options
- Common Commands
- Generating and Modifying Configuration Files - config
- Creating a Bucket - mb
- Deleting a Bucket - rb
- Tagging Bucket - bucket-tagging
- Querying Bucket/Object List - ls
- Obtaining Statistics on Different Types of Objects - du
- Uploading/Downloading/Copying Objects - cp
- Syncing Upload/Download/Copy - sync
- Deleting Objects - rm
- Getting File Hash Value - hash
- Listing Incomplete Multipart Uploads - lsparts
- Clearing Incomplete Multipart Uploads - abort
- Restoring Archived Objects - restore
- Getting Pre-signed URL - signurl
- FAQs
- COSCMD
- COS Migration
- FTP Server
- COSFS
- Hadoop
- COSDistCp
- Hadoop-cos-DistChecker
- HDFS TO COS
- Diagnostic Tool
- GooseFS
- Overview
- Update Log
- Getting Started
- Key Features
- Deployment Guide
- OPS Guide
- Data Security
- GooseFS on TKE Cloud Native Practices
- Quick Start
- Using GooseFS as Fluid Dataset Runtime
- Multi-Master Node Deployment
- Multiple-Master Affinity Scheduling
- Accelerating Data in COS
- Global Client Deployment
- Using Parameters for Encryption
- Manual Scaling
- Data Caching and Metadata Caching
- Data Affinity Scheduling
- Data Scheduling Based on Tolerations and Taints
- Data Preloading
- Deploying Multiple Datasets on the Same Cluster via Placement
- Best Practice
- Overview
- Access Control and Permission Management
- ACL Practices
- Cloud Access Management Practices
- Granting Sub-accounts Access to COS
- Cases of Permission Setting
- Working with COS API Authorization Policies
- Security Guidelines for Using Temporary Credentials for Direct Upload from Frontend to COS
- Generating and Using Temporary Keys
- Grant Sub-account Permissions to Pull Tag List
- Descriptions and Use Cases of Condition Keys
- Granting Sub-account Under One Root Account Permission to Manipulate Buckets Under Another Root Account
- Performance Optimization
- Data Migration
- Accessing COS Using the AWS S3 SDK
- Data Disaster Recovery and Backup
- Direct Data Upload
- Domain Name Management Practice
- Data Security
- Data Verification
- Big Data Practice
- Custom Processing
- Using COS in the Third-party Applications
- Using APIs to Zip Files
- Using APIs to Merge Files
- Developer Guide
- Creating Request
- Bucket
- Object
- Data Management
- Data Disaster Recovery
- Data Security
- Cloud Access Management
- Batch Operation
- Global Acceleration
- Data Processing
- Metadata Acceleration
- Data Workflow
- Monitoring and Alarms
- Troubleshooting
- API Documentation
- Introduction
- Common Request Headers
- Common Response Headers
- Error Codes
- Request Signature
- Operation List
- Service APIs
- Bucket APIs
- Basic Operations
- Access Control List (acl)
- Cross-Origin Resource Sharing (cors)
- Lifecycle
- Bucket Policy (policy)
- Hotlink Protection (referer)
- Tag (tagging)
- Static Website (website)
- Intelligent Tiering
- Bucket inventory(inventory)
- Versioning
- Cross-Bucket Replication（replication）
- Log Management(logging)
- Global Acceleration (Accelerate)
- Bucket Encryption (encryption)
- Custom Domain Name (Domain)
- Object APIs
- Batch Operation APIs
- Data Processing APIs
- Image Processing
- Basic Image Processing
- Scaling
- Cropping
- Rotation
- Converting Format
- Quality Change
- Gaussian Blurring
- Adjusting Brightness
- Adjusting Contrast
- Sharpening
- Grayscale Image
- Image Watermarking
- Text Watermarking
- Obtaining Basic Image Information
- Obtaining Image EXIF
- Obtaining Image’s Average Hue
- Removing Image Metadata
- Quick Thumbnail Template
- Limiting Output Image Size
- Pipeline Operators
- Image Advanced Compression
- Persistent Image Processing
- Image Compression
- Basic Image Processing
- Media Processing
- Image Processing
- Data Workflow APIs
- SDK Documentation
- SDK Overview
- Preparations
- Android SDK
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Restoring Archived Objects
- Querying Object Metadata
- Generating Pre-Signed URLs
- Configuring Preflight Requests for Cross-origin Access
- Server-Side Encryption
- Single-Connection Bandwidth Limit
- Extracting Object Content
- Getting Started
- Quick Experience
- Bucket Operations
- Remote Disaster Recovery
- Data Management
- Cloud Access Management
- Data Verification
- Image Processing
- Setting Custom Headers
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Object Operations
- C SDK
- C++ SDK
- .NET(C#) SDK
- Getting Started
- Bucket Operations
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Checking Whether Objects Exist
- Restoring Archived Objects
- Querying Object Metadata
- Object Access URL
- Getting Pre-Signed URLs
- Configuring Preflight Requests for Cross-Origin Access
- Server-Side Encryption
- Single-URL Speed Limits
- Extracting Object Content
- Cross-Region Disaster Recovery
- Data Management
- Cloud Access Management
- Image Processing
- Setting Custom Headers
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Backward Compatibility
- Go SDK
- iOS SDK
- Getting Started
- Quick Experience
- Bucket Operations
- Object Operations
- Uploading Objects
- Downloading Objects
- Listing Objects
- Copying and Moving Objects
- Extracting Object Content
- Checking Whether an Object Exists
- Deleting Objects
- Restoring Archived Objects
- Querying Object Metadata
- Server-Side Encryption
- Object Access URL
- Generating Pre-Signed URL
- Configuring CORS Preflight Requests
- Cross-region Disaster Recovery
- Data Management
- Cloud Access Management
- Image Processing
- Setting Custom Headers
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Java SDK
- Getting Started
- FAQs
- Bucket Operations
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Checking Whether Objects Exist
- Querying Object Metadata
- Modifying Object Metadata
- Object Access URL
- Getting Pre-Signed URLs
- Restoring Archived Objects
- Server-Side Encryption
- Client-Side Encryption
- Single-URL Speed Limits
- Extracting Object Content
- Data Management
- Cross-Region Disaster Recovery
- Cloud Access Management
- Image Processing
- Media Processing
- Troubleshooting
- Setting Access Domain Names (CDN/Global Acceleration)
- JavaScript SDK
- Node.js SDK
- PHP SDK
- Python SDK
- Getting Started
- Bucket Operations
- Bucket Management
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Checking Whether Objects Exist
- Querying Object Metadata
- Modifying Object Metadata
- Object Access URL
- Getting Pre-Signed URLs
- Restoring Archived Objects
- Extracting Object Content
- Server-Side Encryption
- Client-Side Encryption
- Single-URL Speed Limits
- Data Management
- Cloud Access Management
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Image Processing
- Mini Program SDK
- Error Codes
- FAQs
- COS Service Level Agreement
- Glossary
- Release Notes and Announcements
- Product Introduction
- Purchase Guide
- Getting Started
- Console Guide
- Console Overview
- Bucket Management
- Bucket Overview
- Creating a Bucket
- Deleting a Bucket
- Querying Buckets
- Emptying a Bucket
- Setting Access Permission
- Setting Bucket Encryption
- Setting Hotlink Protection
- Setting Origin-Pull
- Setting CORS
- Setting Versioning
- Setting Up a Static Website
- Setting Lifecycle
- Setting Logging
- Accessing Bucket List Using Sub-Account
- Adding a Bucket Policy
- Setting Log Analysis
- Setting INTELLIGENT TIERING
- Setting Inventory
- Domain Name Management
- Setting Bucket Tags
- Setting Cross-Bucket Replication
- Enabling Global Acceleration
- Object Management
- Uploading Objects
- Downloading Objects
- Copying Objects
- Viewing Object Information
- Searching for Objects
- Sorting and Filtering Objects
- Direct Upload to ARCHIVE
- Modifying Storage Classes
- Deleting Incomplete Multipart Uploads
- Setting Object Access Permission
- Setting Object Encryption
- Custom Headers
- Deleting Objects
- Restoring Archived Objects
- Folder Management
- Data Extraction
- Setting Object Tags
- Exporting Object URLs
- Restoring a Historical Object Version
- Batch Operation
- Monitoring Reports
- Data Processing
- Data Processing Workflow
- Application Integration
- User Tools
- Tool Overview
- Installation and Configuration of Environment
- COSBrowser
- COSCLI (Beta)
- COSCLI Overview
- Download and Installation Configuration
- Common Options
- Common Commands
- Generating and Modifying Configuration Files - config
- Creating a Bucket - mb
- Deleting a Bucket - rb
- Tagging Bucket - bucket-tagging
- Querying Bucket/Object List - ls
- Obtaining Statistics on Different Types of Objects - du
- Uploading/Downloading/Copying Objects - cp
- Syncing Upload/Download/Copy - sync
- Deleting Objects - rm
- Getting File Hash Value - hash
- Listing Incomplete Multipart Uploads - lsparts
- Clearing Incomplete Multipart Uploads - abort
- Restoring Archived Objects - restore
- Getting Pre-signed URL - signurl
- FAQs
- COSCMD
- COS Migration
- FTP Server
- COSFS
- Hadoop
- COSDistCp
- Hadoop-cos-DistChecker
- HDFS TO COS
- Diagnostic Tool
- GooseFS
- Overview
- Update Log
- Getting Started
- Key Features
- Deployment Guide
- OPS Guide
- Data Security
- GooseFS on TKE Cloud Native Practices
- Quick Start
- Using GooseFS as Fluid Dataset Runtime
- Multi-Master Node Deployment
- Multiple-Master Affinity Scheduling
- Accelerating Data in COS
- Global Client Deployment
- Using Parameters for Encryption
- Manual Scaling
- Data Caching and Metadata Caching
- Data Affinity Scheduling
- Data Scheduling Based on Tolerations and Taints
- Data Preloading
- Deploying Multiple Datasets on the Same Cluster via Placement
- Best Practice
- Overview
- Access Control and Permission Management
- ACL Practices
- Cloud Access Management Practices
- Granting Sub-accounts Access to COS
- Cases of Permission Setting
- Working with COS API Authorization Policies
- Security Guidelines for Using Temporary Credentials for Direct Upload from Frontend to COS
- Generating and Using Temporary Keys
- Grant Sub-account Permissions to Pull Tag List
- Descriptions and Use Cases of Condition Keys
- Granting Sub-account Under One Root Account Permission to Manipulate Buckets Under Another Root Account
- Performance Optimization
- Data Migration
- Accessing COS Using the AWS S3 SDK
- Data Disaster Recovery and Backup
- Direct Data Upload
- Domain Name Management Practice
- Data Security
- Data Verification
- Big Data Practice
- Custom Processing
- Using COS in the Third-party Applications
- Using APIs to Zip Files
- Using APIs to Merge Files
- Developer Guide
- Creating Request
- Bucket
- Object
- Data Management
- Data Disaster Recovery
- Data Security
- Cloud Access Management
- Batch Operation
- Global Acceleration
- Data Processing
- Metadata Acceleration
- Data Workflow
- Monitoring and Alarms
- Troubleshooting
- API Documentation
- Introduction
- Common Request Headers
- Common Response Headers
- Error Codes
- Request Signature
- Operation List
- Service APIs
- Bucket APIs
- Basic Operations
- Access Control List (acl)
- Cross-Origin Resource Sharing (cors)
- Lifecycle
- Bucket Policy (policy)
- Hotlink Protection (referer)
- Tag (tagging)
- Static Website (website)
- Intelligent Tiering
- Bucket inventory(inventory)
- Versioning
- Cross-Bucket Replication（replication）
- Log Management(logging)
- Global Acceleration (Accelerate)
- Bucket Encryption (encryption)
- Custom Domain Name (Domain)
- Object APIs
- Batch Operation APIs
- Data Processing APIs
- Image Processing
- Basic Image Processing
- Scaling
- Cropping
- Rotation
- Converting Format
- Quality Change
- Gaussian Blurring
- Adjusting Brightness
- Adjusting Contrast
- Sharpening
- Grayscale Image
- Image Watermarking
- Text Watermarking
- Obtaining Basic Image Information
- Obtaining Image EXIF
- Obtaining Image’s Average Hue
- Removing Image Metadata
- Quick Thumbnail Template
- Limiting Output Image Size
- Pipeline Operators
- Image Advanced Compression
- Persistent Image Processing
- Image Compression
- Basic Image Processing
- Media Processing
- Image Processing
- Data Workflow APIs
- SDK Documentation
- SDK Overview
- Preparations
- Android SDK
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Restoring Archived Objects
- Querying Object Metadata
- Generating Pre-Signed URLs
- Configuring Preflight Requests for Cross-origin Access
- Server-Side Encryption
- Single-Connection Bandwidth Limit
- Extracting Object Content
- Getting Started
- Quick Experience
- Bucket Operations
- Remote Disaster Recovery
- Data Management
- Cloud Access Management
- Data Verification
- Image Processing
- Setting Custom Headers
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Object Operations
- C SDK
- C++ SDK
- .NET(C#) SDK
- Getting Started
- Bucket Operations
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Checking Whether Objects Exist
- Restoring Archived Objects
- Querying Object Metadata
- Object Access URL
- Getting Pre-Signed URLs
- Configuring Preflight Requests for Cross-Origin Access
- Server-Side Encryption
- Single-URL Speed Limits
- Extracting Object Content
- Cross-Region Disaster Recovery
- Data Management
- Cloud Access Management
- Image Processing
- Setting Custom Headers
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Backward Compatibility
- Go SDK
- iOS SDK
- Getting Started
- Quick Experience
- Bucket Operations
- Object Operations
- Uploading Objects
- Downloading Objects
- Listing Objects
- Copying and Moving Objects
- Extracting Object Content
- Checking Whether an Object Exists
- Deleting Objects
- Restoring Archived Objects
- Querying Object Metadata
- Server-Side Encryption
- Object Access URL
- Generating Pre-Signed URL
- Configuring CORS Preflight Requests
- Cross-region Disaster Recovery
- Data Management
- Cloud Access Management
- Image Processing
- Setting Custom Headers
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Java SDK
- Getting Started
- FAQs
- Bucket Operations
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Checking Whether Objects Exist
- Querying Object Metadata
- Modifying Object Metadata
- Object Access URL
- Getting Pre-Signed URLs
- Restoring Archived Objects
- Server-Side Encryption
- Client-Side Encryption
- Single-URL Speed Limits
- Extracting Object Content
- Data Management
- Cross-Region Disaster Recovery
- Cloud Access Management
- Image Processing
- Media Processing
- Troubleshooting
- Setting Access Domain Names (CDN/Global Acceleration)
- JavaScript SDK
- Node.js SDK
- PHP SDK
- Python SDK
- Getting Started
- Bucket Operations
- Bucket Management
- Object Operations
- Uploading Objects
- Downloading Objects
- Copying and Moving Objects
- Listing Objects
- Deleting Objects
- Checking Whether Objects Exist
- Querying Object Metadata
- Modifying Object Metadata
- Object Access URL
- Getting Pre-Signed URLs
- Restoring Archived Objects
- Extracting Object Content
- Server-Side Encryption
- Client-Side Encryption
- Single-URL Speed Limits
- Data Management
- Cloud Access Management
- Setting Access Domain Names (CDN/Global Acceleration)
- Troubleshooting
- Image Processing
- Mini Program SDK
- Error Codes
- FAQs
- COS Service Level Agreement
- Glossary
Practice Scenario
If you have a local IDC, COS offers the following migration methods to help you quickly migrate massive amounts of local data to COS.
| Migration Method | Description |
|---|---|
| COS Migration(online migration) | COS Migration is an all-in-one tool integrating COS data migration features. You can migrate data to COS quickly after completing simple configurations. |
| Cloud Data Migration (CDM) (offline migration) | CDM uses a dedicated offline migration device provided by Tencent Cloud to help you migrate your local data to the cloud, solving the problems that arise from online migration from a local IDC to cloud, such as long time, high costs, and low security. |
You can determine how to migrate data based on the data volume, IDC egress bandwidth, IDC idle server resources, acceptable completion time, and other factors. The estimated time needed for online migration is shown in the figure below. As can be seen, if data migration needs to take more than 10 days or the amount of data to be migrated exceeds 50 TB, you are recommended to use CDM for offline migration; otherwise, online migration is a better choice.
Data migration may be affected if there is a high number of small files below 1 MB or the performance of disk IO is insufficient.
Migration Practice
COS Migration
The steps are as follows:
- Install the Java environment.
- Install the COS Migration tools.
- Modify the configuration file.
- Launch the tool.
For more information, please see COS Migration Tool.
Tips
Below describes how to configure COS Migration to maximize the migration speed:
- Adjust the threshold and concurrence of large/small files based on your network environment to implement optimal migration where large files are split into multiple parts and small files are transferred concurrently. Adjust the tool execution time and set a bandwidth limit to ensure that bandwidth usage by data migration will not affect your business operations. Such adjustments can be made by modifying the following parameters in the
[common]section of theconfig.iniconfiguration file:Parameter Name Parameter Description smallFileThreshold Threshold for small files. If the size of a file is higher than or equal to this threshold, multipart upload will be used; otherwise, simple upload will be used. The default value is 5 MB. bigFileExecutorNum Concurrence of large files, which is 8 by default.
If COS is connected to over the public network with low bandwidth, reduce this value.smallFileExecutorNum Concurrence of small files, which is 64 by default.
If COS is connected to over the public network with low bandwidth, reduce this value.executeTimeWindow This parameter defines the time range when the migration tool will execute a task every day, which will enter sleep mode at other times. In sleep mode, the migration will be paused and the migration progress will be retained until the next time window when the migration will be resumed automatically. - Distributed parallel transfer can further speed up the migration. You can install COS Migration on multiple servers and perform migration tasks separately for different data sources.
Cloud Data Migration (CDM)
The steps are as follows:
- Go to the CDM Console to submit an application.
- After the application is approved, you can wait to receive a device.
- After receiving the device, copy your data to it as instructed in the migration device user manual.
- Then, submit a return application in the console and wait for Tencent Cloud to migrate your data to COS.
For more information, please see Cloud Data Migration (CDM).
Tips
Below describes how to migrate data to COS offline efficiently and securely.
- Configure a 10 Gbps network connection for your IDC. To remove potential constraints of the local data environment on data transfer, you can use a high-performance server as a mount point to maximize the replication speed.
- Parallel transfer is the fastest way to transfer data though CDM. You can monitor the CPU and memory utilization of the device and if the current migration speed is lower than expected, you can speed the migration up in the following ways:
- Connect multiple devices to the same CDM device via different network interfaces.
- Connect multiple devices to several CDM devices via different network interfaces.
Yes
No
Was this page helpful?