This document introduces a free-of-maintenance approach to import Kafka data to Cloud Data Warehouse PostgreSQL instances by using SCF.
Cloud Data Warehouse PostgreSQL (CDWPG) can sync messages from the messaging middleware for analysis.
Limits
Only Tencent Cloud CKafka is supported as the data source. External Kafka services are not supported.
One function can only import data to one table in CDWPG. To write data into multiple tables, you need to create one function for each table.
Directions
Step 1. Create a function
In the SCF console, select Functions > Create. In the Create page, enter ckafka and CDW in the Fuzzy search field, complete the settings and click Next.
On the Function configuration page, complete the settings in Environment configuration and Network configuration in Advanced configuration as follows:
Environment configuration
Memory: Set the memory based on the actual running status, which is 128 MB by default. If it is insufficient during data import, you should increase it.
Environment variable:
Parameter
Required
Description
DB_DATABASE
Supported
Database name
DB_HOST
Supported
If the function is deployed in a VPC and in the same subnet as CDWPG, you can enter the private IP of CDWPG; otherwise, enter the public IP and configure an allowlist.
DB_USER
Supported
Username
DB_PASSWORD
Supported
User password
DB_SCHEMA
Supported
Schema name. If it is not specified during table creation, it will be `public` in general.
DB_TABLE
Supported
Table name
DB_PORT
No
CDWPG port, which is 5436 by default.
MSG_SEPARATOR_ASCII
No
ASCII code of the data delimiter in CKafka, which is 39 (comma) by default. As commas usually show up in the business data, we recommend you set this parameter to 11 (vertical bar).
MSG_NULL
No
NULL value of CKafka consumption. The default value is `\\N`
REPLACE_0X00
No
Whether to replace "0x00" in strings. The default value is 0 (1 indicates to replace).
ENABLE_DEBUG
No
Whether to print error records. The default value is 0 (1 indicates to print).
ENABLE_COS
No
Whether to dump unwritten records to COS. The default value is 0 (1 indicates to dump).
COS_SECRET_ID
No
`secret_id` for COS access. If `ENABLE_COS` is 1, this field is required.
COS_SECRET_KEY
No
`secret_key` for COS access. If `ENABLE_COS` is 1, this field is required.
COS_BUCKET
No
COS bucket name. If `ENABLE_COS` is 1, this field is required.
STATMENT_TIMEOUT
No
Query timeout period, which is 50 seconds by default.
Network configuration
VPC: Activate VPC and set the same VPC and subnet values as those of the CDWPG instance.
The corresponding values in CDWPG are as shown below:
Public Network Access: Enable
Step 2. Configure a trigger
In the Functions list in the SCF console, click the name of the newly created function to enter the function details page and click Trigger management > Create trigger on the left to create a trigger. Here, set CKafka trigger for Trigger method.
