Setats Cluster
A Setats service instance, which is independently deployed and managed, consists of Manager and Worker nodes. All user data read/write and management operations are performed within the cluster.
Manager
The management node in a Setats cluster is responsible for global metadata management, bucket scheduling, and node coordination. Manager supports multi-replica HA deployment. The primary node (Active) and standby node (Standby) achieve automatic failover through state synchronization, ensuring uninterrupted cluster management services.
Worker
The worker node in a Setats cluster handles actual data read/write and storage at the granularity of buckets. Each Worker manages several buckets and is responsible for data writing, merging, caching, and persistence.
Database (DB)
A logical unit that organizes, stores, and manages data according to its data structure under a cluster. Users can create, update, or delete tables within a database. A database is used to group and manage tables, following a hierarchical structure: cluster > database > table.
Table
A table is the basic unit for storing user data in Setats, organized in rows and columns. Tables are stored in specific databases. Setats supports primary key-based INSERT, UPDATE, and DELETE operations, as well as various Upsert semantics such as Partial Update (partial column update) and Aggregation (pre-aggregation).
Partition
Based on user-specified partition keys, table data is logically divided into smaller, more manageable subsets. This mechanism supports flexible partitioning by dimensions such as time and business, facilitating data lifecycle management and query pruning. A distinct partition is defined by each unique value (or combination of values) in the partition column. When no partition column is defined, all data in the table belongs to a single default partition.
Bucket
Within a partition, data is further horizontally split into multiple buckets based on primary key hashing. A bucket is the smallest granularity for data distribution, scheduling, and migration in Setats. The number of buckets can be configured per table. Each bucket independently manages its own data files and indexes and is handled by a Worker node.
Buffer
Each bucket maintains a data buffer locally on the Worker, which stores recently written hot data in row-based format. The buffer handles efficient queries on historical records during immediate merging and also supports low-latency primary key point lookups and real-time streaming reads. Data in the buffer is periodically asynchronously written to remote storage for persistence.
Remote Data File
Data in the buffer is periodically flushed to remote storage (COS/HDFS) and persisted as data files in columnar format (Parquet). These files store the table's full historical data and are suitable for offline batch analysis and scan query scenarios.
Changelog
Setats automatically generates a change log during data updates. Each data change produces records containing the full-field UpdateBefore (pre-update snapshot) and UpdateAfter (post-update snapshot), completely recording the insert, update, and delete operations for each piece of data. Downstream stream processing engines, such as Flink, can subscribe to the Changelog in real time for incremental processing.
Delete Vector
Setats adopts a Merge-on-Write policy. During writes, it locates the storage location of old version records through a Lookup operation and generates a deletion vector to mark them as invalid. During queries, only the invalidated records need to be filtered based on the deletion vector, eliminating the need to perform time-consuming merge operations and significantly reducing the overhead on the read side.