tencent cloud

PrevNext

Feedback

search by keyword

Recent Pages

Documentation
TDMQ for CKafka
    Documentation
    Download PDF

    Connection Management

    Last updated: 2024-01-09 14:54:11
    Download PDF

      Overview

      When a large number of data tasks share the same data source or target, the configuration information of the source or target needs to be entered repeatedly each time a task is created, which is time-consuming.
      With CKafka Connector, you can create separate connections. After a connection is created, it can be directly associated with a specific data task as a data source or target with no repeated configurations needed, reducing your operating costs.
      A connection can be associated with multiple data tasks. Currently supported connection types include ClickHouse, CTSDB, Doris, Data Transmission Service (DTS), Elasticsearch Service (ES), MongoDB, MySQL, PostgreSQL, MySQL, MariaDB, and SQL Server.
      This document describes how to create, modify, and delete a connection in CKafka Connector.

      Directions

      Creating a connection

      1. Log in to the CKafka console.
      2. Select Connector > Connection List on the left sidebar, select the region, and click Create Connection.
      3. Select the connection type in the pop-up window, click Next, and enter the connection configuration information.
      ClickHouse
      CTSDB
      Doris
      DTS
      ES
      MongoDB
      MySQL
      PostgreSQL
      TDSQL-C
      MariaDB
      SQL Server
      CDWPG
      TDSQL for PostgreSQL
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Data Warehouse Type: Select Cloud Data Warehouse (CDWCH) or Self-built ClickHouse.
      CDWCH: As an instance has been encapsulated with a private connection during creation, you can directly select the corresponding CDWCH instance in the console, and the data distribution feature will automatically connect to the instance's VPC.
      Self-built ClickHouse: As the CKafka instance is a managed instance and EMR ClickHouse creates a public network route on the purchased CVM instance directly, you need to manually create a CLB instance to connect to the VPC. The following steps use EMR ClickHouse as an example to create a CLB instance:
      1.1.1 Go to the EMR console, select the target cluster, click Cluster Resource > Node Status, and find the ClickHouse node IP on the status page.
      1.1.2 Go to the CLB console, create a CLB instance, click Listener Management on the top, click TCP/UDP/TCP SSL Listener on the page, and enter the port used during data distribution as the port.
      1.1.3 After creating a listener, click Bind Backend Service and enter the TCP port of ClickHouse, which is 9000 by default.
      1.1.4 After the binding, you can select the created CLB instance and enter the port listened on by the CLB instance on the data distribution page in the CKafka console.
      Note:
      Currently, you can only create a data distribution to ClickHouse task in the same region as the CLB instance.
      Username: ClickHouse username, which is default by default.
      Password: ClickHouse password.
      For security reasons, the password is required. Currently, the password may be empty after an instance is created, in which case you need to modify the password in the user.xml configuration file. For detailed directions, see the ClickHouse documentation.
      
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      CTSDB Address: Enter the address of the connected CTSDB database.
      Username: Enter the database username.
      Password: Enter the database password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Source Database Type: The default option is Self-built/EMR Doris.
      CLB Instance: Only private network CLB instance is supported, and the Doris FE and BE ports must be simultaneously mounted to the CLB instance.
      FE Port: FE JDBC port, which defaults to 9030.
      BE Port: BE HTTP port, which defaults to 8040.
      Username: Enter the database username.
      Password: Enter the database password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      DTS Instance: Select a DTS instance. The partition count of the topics that are subscribed to in DTS must be set to the same as that of the target Kafka topics.
      DTS Consumer Group: Select a DTS consumer group.
      Consumer Group Account: DTS consumer group account.
      Consumer Group Password: DTS consumer group password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      ES Instance Cluster: Select an ES instance cluster.
      Instance Username: Enter the ES instance username, which cannot be modified once set. It is elastic by default.
      Instance Password: Enter the ES instance password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Source Database Type:
      TencentDB for MongoDB: Select a database instance.
      Self-built MongoDB: Select your CLB instance and specify the port.
      Username: Source MongoDB database username.
      Password: Source MongoDB database password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Source Database Type:
      TencentDB for MySQL: Select a database instance.
      Self-built MySQL: Select your CLB instance and specify the port. Only private network CLB instance is supported. For CLB-based data integration, the CLB instance can only be mounted to one source database due to the restrictions of MySQL's sync mechanism.
      Username: Enter the MySQL username.
      Password: Enter the MySQL password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Source Database Type:
      TencentDB for PostgreSQL: Select a database instance.
      Self-built PostgreSQL: Select your CLB instance and specify the port. Only private network CLB instance is supported. For CLB-based data integration, the CLB instance can only be mounted to one primary database due to the restrictions of PostgreSQL's sync mechanism.
      Username: Enter the PostgreSQL username.
      Password: Enter the PostgreSQL password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Database Type: Select PostgreSQL or MySQL.
      Database Instance: Currently, you can only select a running instance.
      Username: Enter the database username.
      Password: Enter the database password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Database Instance: Select a database instance.
      Username: Enter the database username.
      Password: Enter the database password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Database Instance: Select a database instance.
      Username: Enter the database username.
      Password: Enter the database password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Database Instance: Select a database instance.
      Username: Enter the database username.
      Password: Enter the database password.
      Connection Name: Enter the connection name.
      Description: Enter the optional connection description.
      Database Instance: Select a database instance.
      Username: Enter the database username.
      Password: Enter the database password.
      4. After entering the connection configuration information, click Next to verify the connection. After the verification is passed, the connection is created, and you can see it in the Connection List.

      Editing the configuration

      1. Log in to the CKafka console.
      2. Select Connector > Connection List on the left sidebar and click the ID of the target connection to enter its Basic Info page.
      3. Click Edit Configuration in the top-right corner of the Basic Info module to modify the connection configuration information. You can choose whether to enable Update & Restart All Associated Tasks. If it is enabled, all tasks associated with the connection will be updated and restarted.

      Viewing associated tasks

      1. Log in to the CKafka console.
      2. Select Connector > Connection List on the left sidebar and click the ID of the target connection to enter its Basic Info page.
      3. Select the Associated Task tab at the top of the page to view the list of tasks associated with the connection. You can filter tasks by Data Source or Data Target.

      Deleting a connection

      On the Connection List page, click Delete in the Operation column and click OK in the pop-up window to delete the connection.
      Note:
      A connection can be deleted only if it has no associated tasks.
      
      Contact Us

      Contact our sales team or business advisors to help your business.

      Contact Us
      Technical Support

      Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

      Submit a Ticket
      7x24 Phone Support
      Copyright © 2013-2024 Tencent Cloud. All Rights Reserved.
      Privacy PolicyLegalCookie Policy