Setats implements a new unified storage system that supports multi-level storage and heterogeneous access. It can meet diverse requirements, including offline batch analysis, complete incremental logs, primary key queries, and state storage, using a single unified storage system, and it supports end-to-end second-level data visibility.
The Setats product offers the following core advantages:
Second-Level Data Visibility
Setats offers robust data write and immediate visibility capabilities. The latency from data change to its perception by downstream consumption systems is typically within seconds, significantly reducing the end-to-end delay from data collection to analysis. This capability is particularly critical for business scenarios with stringent timeliness requirements, such as real-time risk control, monitoring and alarm, and recommendation systems, enabling business decisions to be truly driven by the latest data.
Supporting a Complete Changelog Incremental Mechanism
During data updates, Setats automatically generates a complete Changelog (Change Log), which fully records the insert, update, and delete operations for each piece of data, including the full-field UpdateBefore and UpdateAfter. This provides a robust data foundation for downstream stream processing engines like Flink, enabling real-time incremental processing. Based on the complete change records, developers can build multi-level data views, perform chained incremental computations, and achieve low-latency, high-throughput real-time data warehousing and complex event processing.
Supporting Batch Processing and OLAP Queries
Setats builds a multi-level, incrementally constructable data warehouse model. Full data is persisted in columnar format to remote storage, supporting high-performance batch processing and multidimensional analysis (OLAP) queries through compute engines such as Doris, Starrocks, and Spark. Users can perform T+0 real-time analysis and execute complex computations on historical data, truly achieving a unified batch and streaming analytics architecture.
Supporting Multiple Upsert Semantics
Setats internally builds a flexible update logic framework that natively supports multiple update policies, such as Upsert, Partial Update (partial field update), and Aggregation (pre-aggregation). Based on business needs, users can update fields or build aggregated metrics, effectively reducing data processing complexity and improving storage and query efficiency.
High-Performance Data Read and Write
At the core of Setats is a self-developed, high-performance data read/write engine. By optimizing the underlying storage format—using a row-column hybrid storage to achieve hot/cold data tiering and multi-level file indexing to accelerate historical record location—and combining it with a fully asynchronous execution model, the engine enables rapid lookup of previous values and immediate completion of merges upon data arrival. This design allows Setats to synchronously generate complete incremental logs and merged data during writes, thereby achieving second-level visibility for both data and Changelog. It serves as the underlying technical foundation supporting all the aforementioned capabilities.