Product Overview
Download
Modo Foco
Tamanho da Fonte
Setats is a unified stream-lake storage system developed by the Oceanus team. It aims to support multiple processing modes, including batch write/read, stream write/read, and state storage, using a single storage system while achieving second-level data visibility. In traditional big data architectures, data analytics relies on multiple storage systems such as data lakes, message queues, and local KV stores. These systems are isolated from each other, leading to data redundancy, complex Ops, and high costs. By adopting a unified storage architecture, Setats supports various data read/write modes, thereby eliminating the redundancy and complexity introduced by multiple storage systems.
Setats is designed to support the following use cases:
Capability Feature | Detailed Description |
Sub-second streaming write and read | Data becomes visible to downstream systems within seconds after being written, supports real-time change data capture (CDC) subscription, and meets the requirements of low-latency real-time data processing scenarios. |
High-performance batch write and read | Full data is persisted to remote storage (COS/HDFS) in columnar format, supports efficient offline batch analysis and scan queries, and is compatible with traditional data lake usage patterns. |
Cold-hot separation with row-column hybrid storage | Setats natively supports tiered data storage for tables, efficiently stores full data in columnar format remotely, and caches hot data in row format locally to accelerate scenarios such as data compaction and point queries. Setats automatically manages data movement and eviction between local and remote storage, eliminating the need for manual TTL or scheduled flush policy configuration, thereby being transparent to users and balancing the high throughput of offline analysis with the low latency of online queries. |
Point query capability | Supports efficient Flink Lookup Join and, via a customized client, supports high-performance primary key point queries and primary key prefix query capabilities (to be released in a future version). |
State storage | Supports state storage for operators such as Flink Delta Join (to be released in a future version). |
Multi-index acceleration | Supports secondary indexes to accelerate data query performance (to be released in a future version). |
Ecological compatibility | Compatible with common open-source lake formats such as Iceberg (to be released in a future version). |
Product Architecture
Distributed Architecture
Setats employs a classic Manager/Worker distributed architecture. Setats table data is organized using a two-tier structure of partitions (Partition) + buckets (Bucket):
Partition: Data is logically divided based on user-specified partition keys. This mechanism supports flexible partitioning by dimensions such as time and business, facilitating data lifecycle management and query pruning.
Bucket: Within a partition, data is further divided into multiple buckets based on primary key hashing. These buckets serve as the smallest granularity for data distribution and scheduling, and can be used to accelerate scenarios such as primary key queries.
The Manager node is responsible for global metadata management, bucket scheduling, and node coordination. It supports multi-replica deployment, where automatic failover between the primary node (Active) and standby node (Standby) is achieved through state synchronization, ensuring HA for the cluster management plane. The Worker node handles actual data read/write and storage at the bucket granularity. Each bucket independently manages its own data files and indexes, inherently supporting parallel processing and horizontal scaling. Worker nodes can be added or removed on demand based on business load.
Instant Merge-on-Write Data Merging
Setats supports various data write semantics, such as partial column updates. For these writes, new data must be merged with historical records to obtain the complete latest row. Simultaneously, Setats supports generating a complete Changelog, meaning update operations need to produce UpdateBefore and UpdateAfter records containing all fields. This also relies on querying historical data during the write process. Therefore, data merging is a crucial component in the Setats write pipeline.
Traditional data lakes typically adopt a policy of batch writes followed by periodic merging (Merge-on-Read / periodic Compaction). After data is written, it must wait for the background merge scheduling to complete before becoming visible to downstream systems, resulting in visibility delays typically at the minute level or longer. In contrast, Setats requires support for second-level data visibility and therefore employs an immediate data merging strategy: after data is written, a Lookup operation first retrieves the corresponding historical data by primary key, and the new and old data are merged immediately—for example, in a Partial Update scenario, the newly written partial columns are concatenated with the remaining columns from the historical record to form a complete record. Once the merge is complete, the latest complete data is written to an in-memory buffer, while the corresponding Changelog is generated for real-time downstream subscription. Data in the buffer is periodically asynchronously written to remote storage (COS/HDFS) for persistence. The query side achieves second-level data visibility by integrating the latest data from the local buffer with the historical data from remote storage.
For multiple writes of data with the same primary key, Setats adopts an overall Merge-on-Write policy: during the Lookup of historical data, it simultaneously locates the storage position of the old version record and generates a deletion vector (Deletion Vector), marking the old version as invalid. Consequently, during queries, only the invalidated records need to be filtered based on the deletion vector, eliminating the need to perform time-consuming merge operations and significantly reducing the overhead on the read side.
Hot and Cold Separation with Row and Column Hybrid Storage
To support fast Lookup operations, Setats adopts a hybrid row-column storage architecture with hot/cold separation in its storage design to address this challenge.
Full data is persisted to remote storage (COS/HDFS) in a columnar format (such as Parquet), meeting the high-throughput batch scanning requirements of offline analytics scenarios and reducing overall storage costs.
Hot data is cached in local memory and disk in a row storage format, ensuring efficient queries of historical records during immediate merges and supporting low-latency scenarios such as primary key point lookups. When a row storage cache miss occurs, Setats supports performing efficient point lookups directly based on remote columnar storage files. Simultaneously, it loads the hot data into the cache in the background, avoiding the blocking caused by waiting for file format conversion and reducing the overall query latency.
The system further accelerates Lookup performance at the metadata layer through indexes such as LSM Tree, Bloom Filter, and Min-max, along with batch operations.
High-Performance Data Read and Write Engine
To support high-performance data read/write with second-level latency, Setats implements a fully asynchronous, high-performance data read/write engine. During the processing of a single data record, I/O operations (such as reading local or remote files) are not blocked from processing subsequent data. Merge processes for multiple data records can execute concurrently. This maximizes CPU utilization and prevents CPU idle time while waiting for I/O returns. This design is particularly crucial in large-scale immediate merge scenarios, enabling Setats to maintain high write throughput while preserving second-level visibility.
Multi-Engine Integration
Setats provides a unified data storage layer that supports various access patterns, including streaming read/write, batch read/write, and primary key search. Through an open format, Setats supports read/write and query operations for multiple compute engines, such as Flink, Spark, Doris, and StarRocks. Using a unified Catalog and table interface, different compute engines can seamlessly access data within Setats.
Ops Support
The Setats service provides Ops management capabilities, helping users efficiently manage clusters and data:
Monitoring and Alerting: It provides built-in multi-dimensional cluster and table-level monitoring metrics (such as write throughput, read latency, buffer utilization, remote storage read/write volume, and so on), supports integration with mainstream monitoring and alerting platforms, and enables real-time detection and timely alerting for abnormal states.
Setats UI: It provides a visual management interface, supporting the viewing of table metadata, partition and bucket distribution, data volume statistics, as well as the operational status, resource utilization, and bucket allocation of each Worker node. This facilitates Ops personnel in quickly locating issues and assessing cluster health.
Ajuda e Suporte
Esta página foi útil?
Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.
comentários