This guide walks you through the process of rapidly provisioning data services via the TCLake console.
Prerequisites
1. TCLake is currently in closed beta and available by invitation only. You can request access by submitting the application form. 2. Once approved, navigate to the TCLake Console. Click Authorize, then confirm by clicking Agree to Authorize in the pop-up dialog to enable the service. Note:
After you click to activate the service, the system automatically creates a dedicated metadata service resource group for you. The provisioning process takes approximately 10 minutes.
3. Upon activation, the initial provisioning user and the Tencent Cloud main account are automatically granted TCLake Administrator privileges. To prevent access errors, we strongly recommend that standard users (CAM sub-users) wait until the administrator has finalized the initial configuration before accessing the service. For a CAM sub-user to access TCLake and manage data catalogs, the following permissions are required:
3.1 Cloud API Permissions: Standard CAM users require the QcloudTccatalogFullAccess policy.
3.2 Data Permissions: Only the Tencent Cloud main account and users with the TCLake Admin role have permission to create data catalogs. Other users must be granted the necessary data permissions by a TCLake Admin user.
Creating a Data Catalog
Data catalogs are classified into two categories based on their source: built-in data catalogs and external data catalogs. The metadata for built-in catalogs is natively stored within the Tencent Cloud TCLake service, supporting three catalog types: structured data (Lakehouse), unstructured data (Volume), and machine learning models (Model). External data catalogs establish connections to external data sources (e.g., MySQL) via JDBC or similar protocols to retrieve metadata in real time. For more details, see Data Catalogs. |
Built-in Data Catalog | Lakehouse Catalog | A structured data catalog that natively hosts open formats, including the TCIceberg batch-stream unified table format and the Lance multimodal format. |
| Volume Catalog | An unstructured data catalog designed to directly manage files (e.g., images, video, and audio), enabling unified metadata governance for unstructured assets. |
| Model Catalog | A built-in catalog for machine learning model artifacts that registers models trained in MLOps frameworks such as MLflow to TC-Catalog, enabling full lifecycle management of ML models. (Currently under development) |
External Data Catalog | e.g., MySQL, EMR, DLC, TCHouse | Establishes connections to external data sources via JDBC or similar protocols to retrieve their metadata in real time. |
The following section demonstrates how to create a new data catalog, using a Lakehouse catalog as an example.
Creating a Lakehouse Catalog
1. Log in to the TCLake console and click Create Data Catalog on the Data Catalog List page. 2. Select Lakehouse as the catalog type and configure the following parameters:
|
Catalog Name | Required. A unique identifier (no duplicates allowed). Must be 1 to 64 characters long and can only contain letters, digits, and underscores. |
Description | Optional. |
Storage Class | Currently, only the Standard storage class is supported. |
Enable Multi-AZ Redundancy. | Note: This feature is currently in limited release. To request access, please submit a ticket. Disabled by default. When enabled, data is redundantly stored across multiple Availability Zones (AZs) within the same region, providing intra-city disaster recovery capabilities. Once enabled, Multi-AZ redundancy cannot be disabled. While it provides higher data reliability, the storage usage fee will also increase. It is recommended to enable this feature when your data has higher reliability requirements. |
Managing Permissions
The permission system for the multimodal intelligent data lake TCLake consists of Cloud API Permissions and Data Permissions. To access TCLake pages via the console and view metadata in data catalogs, you must pass both layers of permission verification.
Cloud API Permissions: These permissions govern access to all TCLake TencentCloud APIs. They are primarily used for viewing operations in the console and performing metadata operations via TencentCloud APIs. If you lack TCLake TencentCloud API permissions, you can contact a Tencent Cloud main account or a user with the CAM administrator role to grant you the QcloudTccatalogFullAccess (full read/write access to TCLake TencentCloud APIs) or QcloudTccatalogReadOnlyAccess (read-only access to TCLake TencentCloud APIs) permission.
TCLake Data Permissions: TCLake provides fine-grained metadata, data resource usage, and access control across data catalogs. This includes granular access permissions for data catalogs, schemas, data tables, data volumes, and models. It ensures that, even when TencentCloud API permissions are granted, operations on specific metadata and data resources by users must still adhere to more detailed authorization rules, achieving precise data-level protection. You can configure these permissions in the TCLake Console > Permissions & Security module.
Upon service activation, the initial provisioning user and the Tencent Cloud main account automatically become TCLake administrators and are assigned the Admin role.
Users assigned the Admin role in TCLake possess all data permissions and authorization permissions, including the permissions to add and delete roles and to create new catalogs.
To obtain specific data permissions, please contact a user holding the TCLake Admin role.