CKafka Message Backup

Last updated: 2021-04-26 14:19:15

    Overview

    COS introduces the SCF-based CKafka Message Backup feature for users to dump CKafka messages to COS, facilitating data analysis, download, and more.

    Based on the open-source Apache Kafka message queuing engine, Tencent Cloud Kafka (CKafka) provides high-throughput and highly scalable message queuing services. For more information, please see CKafka Product Overview.

    If a backup rule has been configured for a bucket, messages generated in a CKafka instance will be dumped to the COS bucket according to the specified time granularity.

    Precautions

    • If you have added a CKafka message backup rule to your bucket via the COS console, the function will appear in the SCF console. DO NOT delete the function. Otherwise, your rule may not take effect.
    • SCF-available regions support backing up CKafka messages to COS, including Guangzhou, Shanghai, Hong Kong (China), Beijing, Chengdu, Singapore, Mumbai, Toronto, Silicon Valley, and more. For more supported regions, please see SCF Documentation.

    Directions

    1. Log in to the COS console.
    2. In the left sidebar, click Application Integration. Then, find CKafka Message Backup.
    3. Click Configure Backup Rules to go to the configuration page.
    4. Click Add Function.

    Note:

    If you haven’t activated SCF, please go to the SCF console to activate it and authorize the service as instructed.

    1. On the page that is displayed, configure the following items:

      • Function Name: uniquely identifies a function and cannot be modified after being set. You can view the function in the SCF console.
      • Associated Bucket: a COS bucket that stores CKafka messages
      • Time Granularity: You can set an interval (5−15 minutes) to aggregate messages according to the message volume. The dumping performance is affected by the number of files aggregated, the number of partitions, and the value of partition_max. For more information, please see Partition in Glossary.
      • SCF Authorization: (required) SCF needs to be authorized to read messages of CKafka instances and dump the messages to the specified bucket.
    2. Click Next to configure CKafka. The configuration items are as follows:

      • Instance: a CKafka instance as the message source. Only instances in the same region are supported.
      • Topic: a topic used as the message source
      • Start Point: the topic offset for dumping the backup messages
      • Address: a VPC private network address is required. CKafka instances that use a classic network require a routing policy. For more information, please see Adding Routing Policy.

        Note:

        An IP address in the corresponding VPC subnet must be available and support DHCP.

    3. Click Next to configure delivery. The configuration items are described as follows:
      Destination Path: a path to deliver the backup messages. If this field is not specified, the files will be stored in the root directory of the bucket. To use a prefix, end it with a slash (/).

    4. Click Confirm.
      You can perform the following operations on the created function:

      • Click View Log to view the historical running status of the CKafka message backup. If an error is reported, you can click View Log to quickly redirect to the SCF console for viewing the error log details.
      • Click Edit to modify the CKafka message backup rule.
      • Click Delete to delete an unwanted CKafka message backup rule.