V21.1.0
Version Release Notes
Database management
Delay GES computation time
The computed GES is the minimum LES reported by all nodes. The MC subtracts the delay from the minimum LES to define the GES, which is used to advance the absolute delay GES.
Enable nodes to detect their own and other nodes' node types.
When obtaining node information and node routing from MC, return the Tag fields related to node roles, and provide them to sqlengine to determine the nodes to which execution statements need to be forwarded.
Scalability and Performance
Fast Online DDL adds the mode for creating Regions.
Add a parameter that allows instances with pre-planned data distribution to still create Regions rather than new RGs when performing Fast Online DDL.
Replace Into Fully Batched
Replace Into adopts Batch optimization, which is no longer constrained by the table's index type. It can still utilize Batch optimization even when both primary keys and unique indexes exist.
Parallel Granularity Refinement Supports Granularity Division at Different Levels
When parallel granularity partitioning is performed, select SST partitioning/Block partitioning/row-level partitioning based on different data scales.
Automatically and evenly distributes partitioned tables across all nodes during node scaling.
After horizontal node scaling, if there is no MC scheduling for partition balancing, idle nodes may lack partitioned tables. Therefore, MC has added an API for shark to trigger partition balancing after horizontal node scaling. This feature is disabled by default (flush-hyper-nodes-spec-task-enabled), as enabling this policy during scaling for older versions of high-capacity instances could generate massive migration tasks.
Single-partition primary key point lookups do not require obtaining timestamps from MC as a read snapshot.
Single-partition primary key point lookups do not require obtaining timestamps from MC, but instead use UINT64_T as the snapshot to read data. This optimization improves the performance of single-partition primary key point lookups. This feature is enabled by default (tdsql_simple_point_read_no_snapshot).
Optimization of Simple Primary Key Point Lookups
Primary key point lookups do not need to obtain GTS, reducing the number of RPC calls in the execution flow and improving performance.
Added a session-level parameter tdsql_simple_point_read_no_snapshot as the optimization switch.
Removing redundant condition calculations
range queries are optimized as range scans, allowing the corresponding where condition to be removed. This avoids pushdown and redundant condition calculations, improving performance.
Added a session-level parameter enable_range_condition_remove as the optimization switch.
Disaster recovery instances support horizontal scaling of CDC nodes
Disaster recovery instances support horizontal scaling of CDC nodes based on the MySQLClient LogService replay.
Supporting Statement-Level Rollback
When a SQL statement encounters a unique index conflict or a pessimistic lock timeout during execution, the entire transaction no longer needs to be rolled back. Users can re-execute the SQL or execute other SQL to continue reading and writing data within the current transaction. This feature is enabled by default and can be enabled or disabled via the session-level variable tdsql_enable_statement_rollback.
rg merge optimization - asynchronous save snapshot
rg merge originally used a synchronous save snapshot approach, which added approximately 1s of latency. This latency may increase under disk pressure and blocks transactions during the process, leading to increased transaction delays.
After optimization with asynchronous save snapshot, transaction blocking time is reduced by at least 1s, and the total rg merge duration is reduced to the millisecond level, further minimizing the impact on transactions.
Enable the RG merging switch to allow RG merging.
Enabling RG merging allows RGs with identical node distribution and consistent replica roles to be merged; allows empty RGs to be merged into other RGs; and allows indexes to be merged into the RG where the primary key resides.
mrr supports primary keys with variable-length fields.
When the primary key contains variable-length varchar fields, it supports generating query plans for single-table mrr queries, mrr join queries, and batched key access joins. Previously, when the primary key had variable-length varchar fields, these plans were not supported; this restriction has now been lifted.
Check whether peer addr is consistent before updating.
When updating the peerid, perform the update only when it differs from the old id to avoid unnecessary updates.
Making parallel plan selection based on Cost
Previously, parallel plan selection was rule-based, including parallel scanning of tables and tables pushed down to parallel workers; now it has been changed to cost-based (Cost).
Optimize BKA Join generation logic.
Support condition pushdown scenarios in batched RPC-optimized BKA joins.
Syntax and feature
Support querying bulk load transaction information via sql.
Added the system table TDSTORE_BULK_LOAD_TXN_INFO. It can be queried for historical bulk load transactions as well as information about ongoing bulk load transactions. Record explain plan to audit log.
Record the execution explain plan of queries in the audit log to facilitate troubleshooting and root cause analysis.
Support SET PERSIST_ONLY syntax for infrequently used RocksDB-related configuration parameters.
The following parameters now support the SET PERSIST_ONLY syntax. Details:
tdstore_data_db_wal_compression
tdstore_monitor_db_wal_compression
tdstore_raft_db_wal_compression
tdstore_user_cf_default_temperature
tdstore_system_cf_default_temperature
tdstore_raft_cf_default_temperature
tdstore_monitor_cf_default_temperature
tdstore_local_cf_default_temperature
Support the returning syntax.
Support for the returning syntax, such as update returning/insert returning/delete returning/replace returning. To maintain compatibility with oracle, "returning" will be a reserved keyword and can no longer be used for naming tables, columns, and so on.
Added the target_file_size_multiplier_additional parameter to control SST size per level.
Added a target_file_size_multiplier_additional parameter to more granularly control the size of SST files at each level. The calculation method for SST file size at Level N is:
target_file_size_base* target_file_size_multiplier ^ (N - 1) * target_file_size_multiplier_additional[N]
As long as the multiplier coefficient target_file_size_multiplier is fixed at 1, the size of SST files at each level can be flexibly configured via target_file_size_multiplier_additional.
For example, after this parameter is added, the default configuration for user cf is "1:1:1:2:2:4:8". This means when target_file_size_base is set to the default 32M, the SST file sizes for L1~L6 are: 32M, 32M, 64M, 64M, 128M, 256M.
Agent supports reporting resource usage of log nodes.
shark performs dynamic scaling based on the resource usage reported by agents.
Optimization of range locks for RC isolation level
Compatible with native MySQL, range locks are no longer applied under the RC isolation level.
Support for table synchronization feature
Broadcast synchronized tables create a replica on each node in the cluster, and all replicas can provide externally accessible strongly consistent read services.
Log Replica Support
Supports a 2 fully-functional replicas + 1 log replica architecture. Log replicas only store log entries from a specific point to the latest, without containing actual data. When log replicas are in normal status, they provide equivalent availability to 3 fully-functional replicas.
Columnar storage engine allowlist supports TEXT type.
The TEXT type family [TINY/MEDIUM/LONG] TEXT is supported by default in the columnar storage engine.
Stability
BulkLoad supports concurrent submission of raft log.
BulkLoad supports concurrent submission of bulk load commit logs within the same resource group (rg). This prevents transaction accumulation during the commit phase in concurrent submission scenarios, which could affect the stability and performance of data import.
Reinforcement of DDL Rollback Logic
When DDL operations are rolled back to delete data, it must be checked whether the tindex id is identical to prevent accidental deletion of tables with the same name during DDL rollback.
It should be checked whether the replica currently being deleted is the leader.
When executing the DoRemoveReplica task, tdstore needs to check whether it is the leader itself to prevent directly removing the leader.
Fixed an issue where BulkLoad transaction submission triggered a core dump upon leader step down.
BulkLoad encounters leader step down when a transaction is submitted. In such cases, the transaction cannot be rolled back immediately. The current transaction remains in a pending status, and its final status can only be determined when replayed in a new term.
Conflict detection for ddl rename and alter name operations
When concurrently executing Rename table and Create table operations, a conflict check is performed. If conflicts are detected, the latter submitted ddl task will be rejected.
Enhanced memory control capability for binlog log receiver to eliminate OOM
Added the log_receiver_get_raft_log_from_cos_total_size parameter to control the memory usage when the binlog log receiver obtains raft logs from cos. This prevents OOM issues caused by excessive memory consumption when raft logs are fetched from cos.
Account the memory allocated by ha_rocksdb::alloc_key_buffers to table_open_cache_max_mem_size.
The table_open_cache_max_mem_size variable currently tracks the memory allocated on table->mem_root to trigger the memory eviction policy. In the new version, memory allocated by ha_rocksdb::alloc_key_buffers (which is not on mem_root) is also accounted for to precisely calculate memory usage and prevent excessive triggering of the memory eviction policy.
BulkLoad Adaptation for witness
BulkLoad adaptation for witness replicas: witness replicas do not require transferring external sst files.
Full-machine migration provides the Force-destroy feature.
Full-machine migration supports passing the force-destroy option to skip destroy replica tasks on failed nodes, preventing blocking.
Fixed hint parsing issues caused by character set inconsistency between client and system.
When the client's character_set_client and character_set_system character sets are inconsistent, hint settings using set_var+optimizer_switch fail to take effect, affecting hint configuration, outline settings, and outline matching. The issue has been fixed.
Fixed outline matching failure caused by adding semicolons.
When an outline contains a semicolon, it cannot be matched and can only be set without semicolons. This issue has been fixed.
Optimized the issue of excessive memory overhead during bulk load due to periodic calculation of RG approximate size or range statistics.
During bulk load data imports, large batches of SST files may be injected into RocksDB's level-0 tier. During this process, periodic requests for calculating RG approximate size or range statistics would create iterators in the level-0 tier. Due to excessive SST files in level-0, these iterators incurred significant memory overhead, posing an OOM risk. This issue has been optimized: the size estimation logic no longer directly creates iterators but instead adopts a sampling approach, resulting in more stable memory performance.
Fixed the crash caused by the official mem_root_deque lacking a proper move constructor implementation.
The official mem_root_deque lacks a proper move constructor, causing a crash, and the usage of move semantics is risky.
Optimized memory surge issue during bulk load data imports caused by excessive SST files in level-0 tier when iterator data access is enabled
During bulk load data imports, large batches of SST files may be injected into RocksDB's level-0 tier. During this process, if scan requests occur, iterators will be created in the level-0 tier. Due to excessive SST files in level-0, enabling iterators simultaneously opens numerous SST files in the level-0 tier and loads their filter blocks into memory, causing excessive memory consumption. Since the primary memory overhead comes from filter blocks, we have modified the process: SST files generated during bulk load no longer create filter blocks.
Optimized memory overhead for log-only nodes
Adjusted some memory resource parameters based on node types to reduce unnecessary memory overhead for log nodes, primarily:
RocksDB-related: write_buffer_size/num, block cache, and their associated parameters.
raft-related: raft flow control trigger threshold.
SQLEngine no longer broadcasts certain SQL statements to log nodes.
Restoring via Backups
Supported incremental backup concurrency to improve the efficiency of incremental backups.
Before this version, incremental backups were performed by a single background thread, which sequentially traversed the RG leaders on the node every 10 seconds and backed them up one by one (transferring Rlog to the agent, which then uploaded it to cos via file stream). This version implements support for concurrent incremental backups by using a configurable thread pool to enable concurrent Rlog transfers from each RG leader to the agent. New parameters: tdstore_execute_incr_backup_threads: the number of bthreads in the thread pool, tdstore_execute_incr_backup_interval_s: the interval between consecutive incremental backup rounds.
Adjusted Hyper-agent exit logic to prevent loss of incremental backup logs during unexpected agent exits.
hyper-agent did not register signal handlers upon exit, causing in-memory buffered incremental backup Raft logs to be lost during unexpected agent termination. The exit logic has been adjusted to add signal handlers, ensuring Flush operations are performed during shutdown to persist log files from memory to cos, preventing log loss.
Optimized write operations to etcd during full and incremental restores.
During full backups and incremental recovery processes, metadata is stored in cos/nas to reduce write pressure on etcd.
Optimized recovery of MC metadata
The distribution of RGs during metadata recovery needs to consider the disaster recovery mode of cloned instances to meet disaster recovery requirements as much as possible.
Public clouds provide snapshot-based backup and recovery capabilities for CBS.
Full backups are completed by utilizing snapshots based on CBS (Cloud Block Storage) capabilities, and instances are quickly restored using snapshots during rollback operations.
Traditional backup and recovery supports restoring witness nodes.
witness nodes typically have smaller specifications. After traditional backup and recovery are adapted for witness nodes, the solution supports skipping the assignment of ingest sst tasks and raft log replay tasks to witness nodes, guaranteeing successful recovery for log-only 2+1 specification instances.
Security Enhancement
Prevention and Error Reporting of CORRUPT DATA
Added a validation mechanism to the write process to prevent memory overrun and writing of erroneous data. When corrupt data is read, print the complete key-value for troubleshooting and analysis.
Operations
Metrics adaptation for log-only nodes and columnar storage nodes
Adapt metrics under the 2 + 1 log-only architecture, and organize and adjust columnar storage node metrics.
Added a view display for the CLS MysqlClient type to show the real-time status of each connection.
A new system table LOGSERVICE_MYSQL_CLIENT has been added to display the real-time status of each Mysql connection. Added a proactive compaction command to the TDStore client for merging undersized SST files.
This command allows manually initiating compaction to merge undersized SST files, enabling specification of the target SST level, the number of small SSTs to merge, and execution timing (primarily considering manageable load concerns). For example:
compact_sst_merge_small_files --input_level=3 --max_compact_files_count=5000 --max_runtime_sec=600
Support for displaying historical deadlock information via system tables.
Added the system table TDSTORE_PESSIMISTIC_DEADLOCK_INFO. Querying this table on the node performing deadlock detection reveals the occurrence time of all historical deadlocks, the ID of the rolled-back transaction, the IDs of all transactions forming the deadlock cycle, and the ID of the SQLEngine executing each transaction. LogService latency querying and reporting has been changed from replay_leader_ts to commit_ts_barrier.
Adjusted the latency reporting metrics to enhance accuracy, ensuring that the latency of user rg exceeds or equals that of sys rg.
Tool support for the SET PERSIST_ONLY syntax.
Support adding the -read-only option to initiate set persist_only SQL statements for modifying configuration items on eligible nodes.
Bug Fixes
MC may panic when TDStore heartbeats are received during leader switchover.
Large transactions and concurrent active master switchover may cause transactions to be mis-killed.
When a merge region job fails, it may cause pessimistic locks in the corresponding region to stop functioning.
Some participants entered the failed state due to write fence validation failure, while the remaining participants failed to synchronize the prepare Log due to log synchronization failure. Combined with the coordinator's master switchover, this resulted in the coordinator not being recovered and unable to clean up participants in the failed state.
During upgrades, system tables in mysql do not get rebuilt with the change of server_sub_version.
Fixed display issues with columnar virtual tables and potential risks in the columnar compaction code.
Fixed a bug where INSERT IGNORE returned incorrect results when duplicate values are inserted.
Fixed a bug where incorrect parallel task partitioning occurred on remote nodes when group by was fully pushed down.
After an upgrade from an older version to release-21.0.0 is performed, migration operations may occasionally fail and retry indefinitely.
In parallel execution, when a non-correlated subquery returning a single row (not a single value) is executed prematurely, and this subquery serves as an operand for the <=> operator while returning an empty set, the process crashes.
After RG merge, if an expanded RG needs to perform leader switchover, it must check the applied index of the vanished RG and cannot switch leadership to a node where replicas from the vanished RG have logs that are far behind.
When fast online ddl fails, return accurate error messages to the client.
Fixed an issue where CDC nodes failed to obtain the required logs from COS due to backup files not being properly closed when pulling Raft Log.
When rg merge and leader stop are executed concurrently, it may cause the rg state to become abnormal.
Added a parameter to limit the number of temporary files used by SQLEngine, and provided a status variable to monitor current temporary file fd usage.
Hotspot scheduling collects metrics such as cpu and memory usage from peer nodes, using these resource indicators as a basis for scheduling decisions.
Having too many RGs causes MC to report errors when MC enables log-service due to excessive data volume storage.
During the SQLEngine startup process, node information is not persisted promptly. If startup fails for other reasons after successful registration, subsequent restarts attempting to re-register the node will fail due to address conflicts.
alter table now supports changing the auto-increment value in inplace mode.
Optimized error messages for scenarios where DDL submission is restricted (such as during backup phase), providing clearer cause explanations.
When the transaction size exceeds the corresponding memory limit, the error message returned to the client is unclear.
During hotspot scheduling, scheduling only one table or partition at a time generates excessive splitting operations and RGs. These tasks need to be aggregated into a single task to reduce RG splitting.
binlog: When a join update is performed between two tables where columns from one table are used to update another table, the primary key for old rows is incorrectly generated in the binlog.
The current balancing mode tends to split primary key data across multiple RGs. A new scheduling mode is introduced to minimize the splitting of primary key data.
MC LogService scheduling gets stuck due to missing RG information.
The GetRegionsByKeyRange RPC times out due to failure to acquire the HyperNodeInfo lock.
In RocksDB, BottommostCompaction may be triggered if an SST file has no overlap with upper-level SST files and its deletion ratio exceeds the threshold.
In tables with secondary indexes, when the deletion ratio in the top-level files of RocksDB exceeds the threshold, Compaction is triggered for those files, but the columnar storage side fails to properly remove the deletion markers.
The MysqlClient mode used for disaster recovery synchronization will synchronize unexpected system tables.
After histogram statistics for a column (such as c1) are collected, when a query filter condition on the columnar storage node includes 'where c1 is null' or 'where c1 is not null', an execution plan error occurs, resulting in query correctness issues.
In scenarios with columnar partitioned tables, querying the TDSTORE_COLUMNAR_NODE_DATA_COUNT_INFO table causes a core dump.
The MysqlClient used for disaster recovery synchronization introduced a memory limit, resulting in an unexpected slowdown of the synchronization speed.
The MysqlClient used for disaster recovery synchronization introduced a memory limit, causing an unexpected global minimum synchronization timestamp update exception, which resulted in the disaster recovery synchronization becoming stuck.
When a sub-task of fast online ddl fails, the entire task terminates immediately with an error.
When `show create TABLE` discovers an inconsistency between the TABLE and the data dictionary, it throws an error.
After scaling out is performed, balancing operations cause node capacity fluctuations.
After instance upgrade, a node ended up with multiple RGs and a primary RG Tag.
When archiving a task, mc first clears the cache and then persists the data. If persistence fails, the log service will be unable to retrieve this task.
Columnar node migration transfers a large number of RGs at once. When a new RG performs LoadRaftSnapshot, it waits for the parent RG's snapshot index position to reach or exceed the split index. Since the parent RG is also migrated, its local snapshot index does not exist initially, forcing the new RG to wait until the parent migration completes. If the parent RG is exceptionally large, the new RG may experience prolonged waiting until timeout occurs.
After a transaction execution timeout throws an exception, duckdb may fail to roll back the abnormal transaction, causing the next transaction to report an error upon startup due to the detection of an existing transaction.
Several RPCs in tdstore failed to fully return ret in the response, which may cause behavioral errors in sqlengine.
Fixed an issue where MC lock wait timeout caused SQLEngine to repeatedly send CreateDataObjects requests, thereby creating multiple duplicate tasks for the same DDL operation.
During a full backup, if the agent restarts, the files generated during the backup process will not be deleted.
When mc hangs during exit, the leaderless period lasts approximately 30 seconds, which causes sqlEngine to self-terminate.
Currently, if a cos connection fails, it triggers prolonged retry attempts, which may block response packets to SQLEngine, causing hangs or excessive delays in enabling incremental backups. This fix ensures: prolonged retries occur only during throttling (error code 503), while other errors trigger retries for just 20 seconds.
If the brpcstream fails during the initialization phase, it can exit immediately.
When an execution error occurs or the process is killed, it continues to obtain statistical information from storage.
SQL statements forwarded by the Proxy cannot apply statement outlines to use bound execution plans.
When the pk keys are modified across partitions within the same transaction of a partitioned table, a core dump may occur.
When a corrupt data error occurs during secondary index lookup, the log should print the corrupted key-value, but a heap-use-after-free memory overrun occurs at the log printing point.
During online copy, concurrent DML operations collecting table statistics may cause DML waits.
During the initialization of parallel tasks, a failure in a single subtask does not immediately cause the system to report an error.
After split rg or merge rg operations, triggering rg delete immediately may result in partial data loss. It is necessary to wait until the snapshot save for old rg or vanished rg is completed.
fast online ddl may leave residual placeholders uncleaned in mc.
After an rpc failure triggers a connection closure, partial resources on the thd are cleaned up. Subsequent proxy executors attempting to continue processing this state may cause a heap-use-after-free error.
Fixed a memory overrun issue caused by concurrent execution of Batch Put and DDL operations.
Concurrent execution of merge rg and install snaposhot may cause the follow to become stuck.
Batch rename failures may cause a crash.
Long-lived connections executing Prepared Statements may cache excessive ranges.
After statement rollback, the dup flag is not properly set during WBWI rebuild index.
Fixed an issue where parallel queries could crash or cause audit log corruption when audit logging is enabled.
When LogService is deleted, wait for log-recevier to be ready.
When a table is invisible, it is not necessary to update the statistics.
Syntax change
|
Modification | | Added features related to synchronization tables. |
Data Dictionary Change
|
Addition | | This is used to query information about historical Bulk Load transactions and in-progress Bulk Load transactions. |
Addition | | Displays the execution status of each MySQL connection during log replay in MySQL mode. |
Addition | | This system table can be queried on the node performing deadlock detection to obtain the occurrence time of all historical deadlocks, the ID of the rolled-back transaction, the IDs of transactions forming the deadlock cycle, and the Node ID of the SQLEngine executing each transaction. |