tencent cloud

Stream Compute Service

Flink Lookup Join

Download
포커스 모드
폰트 크기
마지막 업데이트 시간: 2026-07-03 15:10:13

Overview

Setats supports efficient Lookup Join, enabling you to use Setats tables as dimension tables. By continuously consuming changelogs and building a local cache, it facilitates real-time join queries between streaming fact tables and dimension tables.

Prerequisites

You have created a Setats cluster and completed the Warehouse configuration.
You have obtained the manager-url.
Dimension table data has been written into Setats and possesses a consumable Changelog.
The Flink runtime environment has completed the preparation of its dependencies.

Dimension Table Creation Syntax

CREATE TABLE `user_dim` (
`id` BIGINT,
`dt` STRING,
`name` STRING,
`age` INT,
`address` STRING,
PRIMARY KEY (`id`, `dt`) NOT ENFORCED
) WITH (
'connector' = 'setats-cdc',
'warehouse' = 'cosn://<bucket>/warehouse',
'catalog-name' = 'setats',
'catalog-type' = 'hive',
'uri' = 'thrift://<metastore-host-1>:7004,thrift://<metastore-host-2>:7004',
'catalog-database' = 'testdb',
'catalog-table' = 'user_dim',
'manager-url' = '<Manager Url>',
'lookup.force-local' = 'true',
'lookup.bucket.discover.interval' = '10s',
'lookup.refresh.async' = 'false',
'lookup.table.store-type' = 'ROCKSDB',
'lookup.table.store-cache-size' = '1024'
);


Lookup Join Parameter Description

Parameter
Description
Default Value
lookup.force-local
Whether to force local Lookup. Currently, set it to `true`.
true
lookup.continuous.discovery-interval
Dimension table Changelog refresh interval
10s
lookup.bucket.discover.interval
New Bucket/partition discovery interval
10s
lookup.refresh.async
Whether to refresh the local dimension table cache asynchronously
false
lookup.table.store-type
Local storage type, with ROCKSDB or HEAP as options
ROCKSDB
lookup.table.store-cache-size
Number of Join Keys that can be stored in the additional Heap Cache in ROCKSDB mode
1024
Usage example:
-- Create a streaming table
CREATE TABLE `order_stream` (
`order_id` BIGINT,
`user_id` BIGINT,
`dt` STRING,
`amount` DECIMAL(10, 2),
`proctime` AS PROCTIME()
) WITH (
'connector' = 'kafka'
-- Omit other Kafka parameters
);
-- Create a Setats dimension table
CREATE TABLE `setats_user_info_lookup` (
`user_id` BIGINT,
`user_name` STRING,
`user_level` INT,
`city` STRING,
PRIMARY KEY (`user_id`) NOT ENFORCED
) WITH (
'connector' = 'setats-cdc',
'warehouse' = 'cosn://<bucket>/warehouse',
'catalog-name' = 'setats',
'catalog-type' = 'hive',
'uri' = 'thrift://<metastore-host-1>:7004,thrift://<metastore-host-2>:7004',
'catalog-database' = 'testdb',
'catalog-table' = 'user_info',
'manager-url' = '<Manager Url>',
'lookup.force-local' = 'true',
'lookup.bucket.discover.interval' = '10s',
'lookup.refresh.async' = 'false',
'lookup.table.store-type' = 'ROCKSDB',
'lookup.table.store-cache-size' = '1024'
);
-- Execute the Lookup Join
SELECT
o.order_id,
o.amount,
u.user_name,
u.user_level,
u.city
FROM `order_stream` AS o
JOIN `setats_user_info_lookup` FOR SYSTEM_TIME AS OF o.proctime AS u
ON o.user_id = u.user_id;


Best Practices

Storage type selection:
ROCKSDB`: Suitable for scenarios with large dimension tables and numerous keys.
`HEAP`: Suitable for scenarios with smaller dimension tables and more latency-sensitive requirements.

도움말 및 지원

문제 해결에 도움이 되었나요?

피드백