Topic Partition

Last updated: 2020-08-04 10:45:10

    Overview

    Partition is the smallest read/write unit of Cloud Log Service (CLS). A log topic can be divided into several partitions and must have at least one partition. CLS uses the value range of MD5 as the valid range and controls the overall throughput performance by merging or splitting partitions. A log topic supports up to 50 partitions. We recommend that you use and work with topic partitions rationally to prevent waste of resources.

    Basic attributes of a partition:

    • Partition number: every partition has a unique number under the same log topic, which is assigned by the system after partition creation or operation.
    • Partition range: every partition has a left-closed and right-open interval.
    • Partition status:
      • Read-write: the current partition allows read and write.
      • Read-only: the current partition is read-only and no data can be written to it.

    Partition Range

    With a partition range, logs can be written in the HashKey mode. The valid range of a log topic is the value range of MD5, which is [00000000000000000000000000000000,ffffffffffffffffffffffffffffffff). All read-write partitions segment the entire value range of a log topic and each occupies a left-closed and right-open interval to ensure that every log record collected is written to the corresponding partition.

    CLS provides two write modes: load-balancing mode and HashKey mode.

    • Load-balancing mode: every data packet is written to a random log topic partition.
    • HashKey mode: every data packet is written to the topic partition that contains the current Key value.

    For example, a log topic has three read-write partitions and their ranges are as follows:

    Partition No. Status Partition Range
    1 Read-write [00000000000000000000000000000000,7fffffffffffffffffffffffffffffff)
    2 Read-write [7fffffffffffffffffffffffffffffff,a0000000000000000000000000000000)
    3 Read-write [a0000000000000000000000000000000,ffffffffffffffffffffffffffffffff)

    If the write mode is HashKey, log data with the Key value of 2fffffffffffffffffffffffffffffff is written to partition 1 and log data with the Key value of 9f000000000000000000000000000000 is written to partition 2.

    Read and Write Capability of Partitions

    Each partition has a certain level of read and write capability. We recommend that you plan partition count based on the actual log traffic of your business. Split partitions when the traffic is higher than the read and write capability of the log topic, and merge partitions when the traffic is lower than the read and write capability of the log topic to save resources.

    Feature Item Description
    Frequency control Write requests Every partition supports a maximum of 500 QPS of write operations. It will reject requests and return the status code 429 with an error message of “SpeedQuotaExceed” when the limit is exceeded.
    Read requests Every partition supports a maximum of 200 QPS of read operations. It will reject requests and return the status code 429 with an error message of “SpeedQuotaExceed” when the limit is exceeded.
    Traffic throttle Write traffic Every partition supports the write traffic up to 5 MB/sec. It will truncate data and return the status code 429 with an error message of “SpeedQuotaExceed” when the limit is exceeded.

    Partition Status

    A partition can be in read-write or read-only mode. Only read-write partitions support data writing. Read-only partitions does not allow data writing but can be consumed within their lifetime. All partitions are readable and writable when they are created, but merging and splitting operations will change the status to read-only.

    Merging partitions

    Two adjacent read-write partitions can be merged into one partition. After the merge, the two original partitions become read-only, which only allow data consumption, but not data writing. The new partition is readable and writable and covers the range of the two original partitions.

    Splitting a partition

    A read-write partition can be split into two partitions with smaller ranges. When splitting a partition, you need to specify the MD5 value of a split point, which must be larger than the value of the start point and smaller than the value of the end point. After the split, the original partition becomes read-only, which only allows data consumption, but not data writing. The new partitions are readable and writable and cover the range of the original partition.

    Was this page helpful?

    Was this page helpful?

    • Not at all
    • Not very helpful
    • Somewhat helpful
    • Very helpful
    • Extremely helpful
    Send Feedback
    Help