Overview

Last updated: 2020-11-13 11:34:15

    Overview

    Cross-bucket replication enables COS to automatically and asynchronously replicate incremental objects from one bucket to another bucket. With cross-bucket replication, COS can accurately replicate the exact same objects, along with object metadata and version IDs, from a source bucket to a destination bucket. Additionally, object operations, such as adding or deleting an object, may also be synced to the destination bucket.

    Note:

    • To enable cross-bucket replication, make sure that versioning is enabled for both the source and destination buckets.
    • After cross-bucket replication is enabled, the object copies will be of the same storage class as the source objects, unless you specify a different storage class during replication.
    • During replication, COS will copy the access control list (ACL) of the source bucket. Currently, the source bucket and destination bucket must be owned by the same account.

    Use Cases

    • Remote disaster recovery: COS boasts 12 9’s of durability for object data, but there is still a slight chance of data loss due to force majeure such as wars and natural disasters. To avoid data loss by explicitly having a separate copy in a different bucket, you can use cross-bucket replication that helps remote disaster recovery. In this way, when the IDC for one bucket is damaged due to force majeure, the IDC for the other bucket can still provide data copies for your use.
    • Compliance: COS ensures data availability by providing multiple copies and erasure codes for data in physical disks by default. However, some industries may have compliance requirements stipulating that you keep copies in another bucket. Cross-bucket replication allows data to be replicated across buckets to meet such requirements.
    • Minimizing access latency: when you have end users accessing objects from different regions, with cross-bucket replication, you can maintain object copies in the buckets closest to them. This minimizes access latency to deliver a better user experience.
    • Special technical requirements: if you have compute clusters in two different buckets and the clusters need to process the same set of data, with cross-bucket replication, you can maintain object copies in both buckets.
    • Data migration and backup: you can copy your business data from one bucket to another as needed for data migration and backup.

    Considerations

    Replication time

    The time it takes for COS to replicate objects between buckets may range from a few minutes to a few hours, depending on factors including:

    • Object size: it takes more time to replicate larger objects, for which multipart upload is recommended for faster upload and synchronization.
    • Distance between the source and destination buckets: A longer distance between the regions of the buckets requires more time to transfer data.
    • How the objects were uploaded: unlike simple upload where data can only be uploaded or downloaded serially, multipart upload supports concurrent uploads which can speed up upload and cross-bucket replication of large files. For more information, please see Simple Upload and Multipart Upload.

    Lifecycle

    Cross-bucket replication requires you to enable versioning first, which will keep multiple historical versions of objects in your buckets and result in more storage usage. COS cross-bucket replication will incur fees for requests, downstream traffic, and storage usage. Among them, storage usage is charged at the prices in the destination bucket’s region. To reduce the costs or customize how to retain your data, use lifecycle management as needed.

    • If you want the object copies in the destination bucket to follow the same lifecycle rule as those in the source bucket, create the same lifecycle rule for the destination bucket.
    • If you have set a lifecycle rule for the destination bucket, note that the creation time of the object copies generated by cross-bucket replication is the creation time of the source objects but not the time when the copies are replicated to the destination bucket.
    • If you have set a lifecycle rule for the source bucket and an object being replicated needs to be deleted according to the lifecycle rule, the object will still be replicated, and the object copy will be retained in the destination bucket.

    Versioning

    To use cross-bucket replication, you need to enable versioning for both the source and destination buckets. For more information, please see Versioning Overview. Once versioning is enabled, you should note that disabling it will affect cross-bucket replication:

    • If you try to disable versioning for a bucket where cross-bucket replication is enabled, COS will return an error prompting you to delete the cross-bucket replication rule before disabling versioning.
    • If you try to disable versioning for a destination bucket, COS will prompt you that doing so will affect cross-bucket replication. If you proceed to disable versioning, the cross-bucket replication rule that uses this bucket as the destination bucket will become invalid.

    Was this page helpful?

    Was this page helpful?

    • Not at all
    • Not very helpful
    • Somewhat helpful
    • Very helpful
    • Extremely helpful
    Send Feedback
    Help