Using COS for Backup and Recovery

Creating a Repository

You can create a repository by running the following command:

PUT _snapshot/my_cos_backup
{
    "type": "cos",
    "settings": {
        "app_id": "xxxxxxx",
        "access_key_id": "xxxxxx",
        "access_key_secret": "xxxxxxx",
        "bucket": "xxxxxx",
        "region": "ap-guangzhou",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "/"
    }
}

• App_id: Tencent Cloud account APPID.
• Access_key_id: Tencent Cloud API key SecretId.
• Access_key_secret: Tencent Cloud API key SecretKey.
• Bucket:COS Bucket name, You can't take your name.-{appId}Suffix .
• Region:COS Bucket region, recommended with ES cluster Intra-region.
• Base_path: backup Directory.

Listing Repository Information

You can use the GET _snapshot To get warehouse information, you can also use the GET _snapshot/my_cos_backup Gets the specified warehouse information.

Creating a Snapshot Backup

Backing up All Indices

Back up all indices in the ES cluster to my_cos_backup Under the warehouse and named snapshot_1 .

PUT _snapshot/my_cos_backup/snapshot_1

This command will be returned immediately and executed asynchronously in Backend Background until the end. If you want to create a snapshot command to block execution, you can add wait_for_completion Parameters. The time of execution of the command is related to the size of the index.

PUT _snapshot/my_cos_backup/snapshot_1?wait_for_completion=true

Backing up the Specified Index

You can specify the index to back up when you create the snapshot. When the value of the parameter indices is multiple indexes, you need to use the,Separated and without spaces.

PUT _snapshot/my_cos_backup/snapshot_2
{
    "indices": "index_1,index_2"
}

Querying a Snapshot

Query individual snapshot information:

GET _snapshot/my_cos_backup/snapshot_1

This command returns information about the snapshot:

{
    "snapshots": [
        {
            "snapshot": "snapshot_1",
            "uuid": "zUSugNiGR-OzH0CCcgcLmQ",
            "version_id": 5060499,
            "version": "5.6.4",
            "indices": [
                "index_1",
                "index_2"
            ],
            "state": "SUCCESS",
            "start_time": "2018-05-04T11:44:15.975Z",
            "start_time_in_millis": 1525434255975,
            "end_time": "2018-05-04T11:45:29.395Z",
            "end_time_in_millis": 1525434329395,
            "duration_in_millis": 73420,
            "failures": [],
            "shards": {
                "total": 3,
                "failed": 0,
                "successful": 3
            }
        }
    ]
}

Deleting the snapshot

Delete the specified snapshot:

DELETE _snapshot/my_cos_backup/snapshot_1

"if there is an incomplete snapshot, the delete snapshot command still executes and cancels the creation process of the incomplete snapshot."

Restoring from a Snapshot

1. Restore all indices backed up in the snapshot to the ES cluster:

POST _snapshot/my_cos_backup/snapshot_1/_restore

• If snapshot_1 includes five indexes, all five indexes will be added to the ES cluster by Resume.
• You can also rename the index using additional options. This option allows you to match the index name through the pattern and provide a new name through the Resume process. You can use this option if you want to validate content or perform other operations on Resume's old data without replacing existing data.

2. Resume single index from the snapshot and provide an alternative name.

POST /_snapshot/my_cos_backup/snapshot_1/_restore
{
    "indices": "index_1",
    "rename_pattern": "index_(.+)",
    "rename_replacement": "restored_index_$1"
}

• Indices: only has Resume index_1 indexes, ignoring other indexes that exist in the snapshot.
• Rename_pattern: looks for the current Resume index on the provided pattern that can be matched.
• Rename_replacement: renames the matching index to an alternative pattern.

Querying Snapshot Restoration Status

Can be implemented by _recovery Command to check the status of snapshot Resume and monitor the progress of snapshot Resume.

1. You can call the following API separately in Resume's specified index.

GET index_1/_recovery

2. This command returns the Resume status of each shard in the specified index.

{
    "sonested": {
        "shards": [
            {
                "id": 1,
                "type": "SNAPSHOT",
                "stage": "INDEX",
                "primary": true,
                "start_time_in_millis": 1525766148333,
                "total_time_in_millis": 8718,
                "source": {
                    "repository": "my_cos_backup",
                    "snapshot": "snapshot",
                    "version": "5.6.4",
                    "index": "index_1"
                },
                "target": {
                    "id": "TlzmxJHwSqyv4rhyQfRkow",
                    "host": "10.0.0.6",
                    "transport_address": "10.0.0.6:9300",
                    "ip": "10.0.0.6",
                    "name": "node-1"
                },
                "index": {
                    "size": {
                        "total_in_bytes": 1374967573,
                        "reused_in_bytes": 0,
                        "recovered_in_bytes": 160467084,
                        "percent": "11.7%"
                    },
                    "files": {
                        "total": 132,
                        "reused": 0,
                        "recovered": 20,
                        "percent": "15.2%"
                    },
                    "total_time_in_millis": 8716,
                    "source_throttle_time_in_millis": 0,
                    "target_throttle_time_in_millis": 0
                },
                "translog": {
                    "recovered": 0,
                    "total": 0,
                    "percent": "100.0%",
                    "total_on_start": 0,
                    "total_time_in_millis": 0
                },
                "verify_index": {
                    "check_index_time_in_millis": 0,
                    "total_time_in_millis": 0
                }
            }
        ]
    }
}

• Type: describes the nature of Resume, the shard is taken from a snapshot of Resume.
• The source: hash describes the specific snapshots and repositories that are the source of Resume.
• Percent: describes the status of Resume. This particular slice has now accounted for 94% of Resume's documents, which is about to be all Resume.

The output lists all the indexes currently in Resume, as well as all the fragments in those indexes. In each fragment, there will be statistical values such as Launch / stop time, duration, Resume percentage, number of transmission bytes and so on.

Cancelling snapshot restoration

DELETE /restored_index_1

If restored _ index If _ 1 is in Resume, the delete command will stop Resume and delete all data that Resume has joined the cluster.