Problem Index
How to Monitor the Running Status of an Instance?
Distributed Cache provides more than 30 monitoring metrics with a minimum collection granularity of 5 seconds, covering the following dimensions:
Resource monitoring: CPU utilization, memory utilization, connection utilization, and bandwidth utilization.
Performance monitoring: QPS (queries per second), command execution latency, and the number of slow queries.
Traffic monitoring: inbound traffic, outbound traffic, and read/write operations per second.
Key monitoring: total number of keys, number of expired keys, and number of evicted keys.
You can also configure alarm policies to automatically send notifications by SMS, email, or WeChat when metrics reach their thresholds. We recommend that you configure at least the following alarms: memory utilization > 80%, CPU utilization > 90%, and connection utilization > 80%.
Additionally, the DBbrain intelligent diagnostics feature can automatically identify performance bottlenecks such as big keys and hot keys, and provide optimization suggestions.
What to Do When Instance CPU Utilization Reaches 100%?
Sustained high CPU utilization is one of the most common performance issues that users encounter. It is usually caused by the following reasons:
High-Complexity Commands
Executing commands with high time complexity, such as KEYS *, SORT, SUNION, and SMEMBERS on large sets, can occupy the CPU for a long time. We recommend that you check slow query logs in the console to identify time-consuming commands and replace them with low-complexity alternatives:
Use SCAN instead of KEYS * for traversal.
Use SSCAN instead of SMEMBERS for large sets.
Avoid running SORT on very large sets. Instead, perform sorting at the business layer.
Concentrated Access to Hot Keys
A large number of requests concentrate on a few keys (hot keys), causing CPU overload on a single data node. You can use the hot Key analysis feature of DBbrain to identify hot keys. Handling methods include:
Implement Local caching for hot keys to reduce the access frequency to Redis.
Split hot keys into multiple subkeys and distribute them across different nodes.
In a cluster architecture, enabling read/write separation distributes read requests to replica nodes.
Massive Short-Term Key Expiration
When many keys expire at the same time, Redis's expiration mechanism consumes additional CPU resources. We recommend that you add a random offset to Key expiration times, such as TTL + random(0, 300), to avoid concentrated expirations.
Excessive Connections
An excessively high number of client connections also increases CPU overhead. Monitor connection utilization in the console. If the number of connections is abnormally high, check whether the client uses a connection pool and whether connection leaks exist.
Troubleshooting Suggestions:
1. In the console, view the CPU utilization monitoring curve to identify the time period when the spike occurred.
2. Check the slow query logs for that period to identify time-consuming commands.
3. Using the intelligent diagnostics feature of DBbrain, the system automatically analyzes the causes of CPU exceptions and provides optimization suggestions.
How to Restore Accidentally Deleted Instance Data?
If instance data is accidentally deleted, for example, by running the FLUSHALL or DEL command, you can restore it by using the following methods:
1. Restore from Backup: In the backup list in the console, select the backup set created before the accidental deletion and restore the data to a new instance. After confirming that the data is complete, switch the application to the new instance.
2. Rollback Restoration (if enabled): If the data rollback feature is enabled, you can roll back instance data to a specified point in time with second-level precision.
Preventive Suggestions:
Disable high-risk commands such as FLUSHALL and FLUSHDB in the console.
Configure multi-account permission control to limit the command execution scope of regular business accounts.
Ensure that the automatic backup policy runs normally, and manually trigger a backup before critical changes.
How to Obtain Client Connection Information and Statistics?
You can use the CLIENT LIST command to view information and statistics about all clients connected to the instance. This command is commonly used to identify connection leaks and confirm connections before a business is taken offline. The key fields returned include:
|
id | Unique 64-bit client ID |
addr | Client address and port |
name | Name set by the client through CLIENT SETNAME |
age | Connected duration in seconds |
idle | Idle duration in seconds |
cmd | Last executed command |
Note:
In high-load and large-scale deployment environments, the CLIENT LIST command may impose significant memory pressure on specific shards. Run this command during off-peak hours.
What to Do When Data Is Inconsistent Between Replica and Master Nodes?
Because Redis uses asynchronous replication, data updates on replica nodes may lag slightly behind those on the master node, which is normal. If the latency remains high, possible causes include:
The I/O write volume of the master node exceeds the synchronization speed of the replica node.
There is network latency between the master node and the replica node.
We recommend troubleshooting through the following methods:
1. View the master-replica synchronization latency monitoring metrics in the console.
2. Check whether frequent writes of big keys cause synchronization buffer overflow.
3. Check whether the network bandwidth of the primary/secondary nodes is sufficient.
If synchronization latency remains high, you can perform a master-replica switchover or restart the replica node in the console to restore synchronization.
What Is the Difference Between Slow Query Logs and Proxy Slow Logs?
Distributed Cache provides two types of slow logs: Proxy layer slow logs and Redis engine layer slow query logs. These two types record different dimensions and must be used together when troubleshooting is performed. For details, see Slow Query. |
Recording location | Redis data node | Proxy layer |
Timing scope | Command execution time only | End-to-end latency including queueing + network forwarding + command execution |
Application scenarios | Locate a single slow command | Identify end-to-end latency issues |
Common Troubleshooting Scenarios:
Proxy slow logs show entries, but Redis engine slow queries show none. This indicates that the latency occurs in the queueing or network layer, possibly because excessive connections or insufficient bandwidth cause request queueing.
Both layers contain records, indicating that the command execution itself takes too long. You need to optimize command complexity or handle big keys.
Redis engine slow queries contain records, but Proxy slow logs contain none. This situation rarely occurs. If it does, contact technical support.
What to Do When Memory Utilization Differs Between Primary and Secondary Nodes?
Differences in memory utilization between primary and secondary nodes are common. Common causes and solutions are as follows:
output buffer Usage
When a replica node synchronizes full data or a client reads a large amount of data, the output buffer temporarily uses additional memory. This is normal behavior, and the buffer is released after synchronization is completed.
Lazy Deletion of Expired Keys
Redis uses a "lazy deletion + periodic scanning" mechanism to remove expired keys. Expired keys on a replica node are not deleted until the master node sends a DEL command. As a result, the replica node may temporarily retain expired keys that have not been cleaned up, causing higher memory usage.
Data Structure Fragmentation
The memory fragmentation rates of primary and secondary nodes may differ. If the fragmentation rate of a replica node is significantly higher than that of the master node, you can try to perform memory defragmentation in the console (the activedefrag parameter).
Recommended Solution
1. Compare the memory utilization and fragmentation rate monitoring of primary and secondary nodes in the console.
2. If the difference is within 10% and tends to remain stable, it is within the normal range and no action is required.
3. If the difference continues to widen, check whether a large number of keys expire at the same time or whether big Key writes exist.
4. If necessary, you can perform a master-replica switchover or restart the replica node to resynchronize data.
What Causes a Primary/Secondary Failover in an Instance and How Does It Affect the Business?
A primary/secondary switchover (also called a master-replica switchover or HA switchover) occurs when the master node of an instance switches to a replica node due to a failure or Ops operations to continue providing services. This is a high availability assurance mechanism of distributed cache databases, ensuring that the instance automatically restores services when the master node becomes abnormal.
Common Causes of Primary/Secondary Failover
|
Master node failure | If the master node process exits unexpectedly, the node goes down, or a heartbeat times out, the system automatically triggers a failover. |
Ops operations | Version upgrade, parameter change (some parameters take effect after restart), scale-out and scale-in, cross-AZ migration |
Manual Trigger | Users can manually perform a primary-replica switch in the console for disaster recovery drills or active migration. |
Resource exception | If the memory or CPU utilization of the master node is too high, a system-protective failover is triggered. |
Impact on Workload
Momentary disconnection: A connection interruption of a few seconds will occur during the switchover, and the client will receive a connection reset or timeout error.
Brief read-only state: A read-only state of up to 1 minute may occur during the switchover while waiting for data synchronization to complete.
Data consistency: Planned switchovers (such as version upgrades, scale-outs, scale-ins, and other Ops operations) complete full and incremental data synchronization before performing the switchover, so no data is lost. Only during unplanned failovers (such as a master node outage), a small number of writes that have not yet been synchronized to replica nodes may be lost because Redis uses asynchronous replication.
Recommendations for Business Adaptation
1. Configure an automatic reconnection mechanism on the client and set a reasonable retry policy (for example, 3 retries at 200 ms intervals).
2. Design write operations to be idempotent to avoid duplicate data caused by retries.
3. Subscribe to instance event alarms in the console to receive primary/secondary switchover notifications in a timely manner.
4. Use connection pooling and configure heartbeat checks to ensure that connections can be quickly restored after they are disconnected.
Why Does Bandwidth Utilization Exceed 100%?
In console monitoring, bandwidth utilization may sometimes exceed 100%. This is not a system exception but a normal phenomenon caused by the statistical method.
Cause Description
Bandwidth utilization statistics for distributed cache databases include two types of traffic:
Business traffic: inbound and outbound traffic generated by client reads and writes.
Master-replica synchronization traffic: internal traffic generated when the master node syncs data to replica nodes.
The bandwidth throttling threshold displayed in the console applies only to business traffic, but the monitoring curve may also include master-replica synchronization traffic. As a result, the monitored value may exceed 100% during full master-replica synchronization or heavy write scenarios.
Impact of Bandwidth Excess
When business traffic does exceed the bandwidth limit of an instance, the traffic control mechanism is triggered:
Inbound limit exceeded: client write requests are throttled, which may manifest as increased response latency or timeouts.
Outbound limit exceeded: client read requests are throttled, and reads of large Values may time out.
Handling method
1. View the separate monitoring of inbound and outbound bandwidth in the console to determine which direction has exceeded the limit.
2. Check whether big Key reads and writes cause instantaneous bandwidth spikes.
3. If your business bandwidth demand does exceed the limit of the current specification, you can adjust the bandwidth in the console (currently free) or upgrade the instance specification.
4. A cluster architecture can distribute traffic by adding shards.
What to Do When Data Skew Occurs in a Cluster Architecture?
Data skew refers to uneven distribution of memory usage or request volume among shards in a cluster, causing some shards to be overloaded while others remain idle.
Identifying Data Skew
On the node management page in the console, view the memory utilization and QPS of each shard:
Memory skew: The memory utilization of a shard is significantly higher than that of other shards (for example, more than 20% higher).
Request skew: The QPS of a shard is much higher than that of other shards.
Common Causes
|
Big Key concentration | Multiple big keys are assigned to the same shard by the Hash algorithm. |
Improper Use of Hash Tags | The service uses {tag} to forcibly route a large number of keys to the same Slot. |
Hot Key | Hot keys are concentrated in a shard, causing request skew. |
Uneven Slot Distribution | In a few scenarios, the Slot data volume is naturally uneven. |
Solution
1. Locate skewed shards: Use node monitoring in the console to find shards with high memory usage or QPS.
2. Troubleshoot big keys: Run a big Key analysis on the skewed shard, and split or clean up the big keys.
3. Optimize Hash Tags: Avoid using the same Hash Tag to force-route many unrelated keys to the same Slot.
4. Distribute hot keys: Split hot keys into different shards by adding random suffixes.
5. Scale out shards: After you increase the number of shards, the system automatically redistributes slots to improve data balance.
How to Troubleshoot Burrs or Sudden Changes in Monitoring Data?
Sudden spikes or drops (glitches) in monitoring curves are usually caused by the following reasons:
Sudden Increase in Memory Utilization
A large number of new Key writes or big Key creations.
Temporary buffer usage during master-replica full synchronization.
A large number of keys are periodically scanned and cleaned up after they expire at the same time, which is reflected as a rise followed by a decline.
Sudden Increase in CPU Utilization
A highly complex command was executed, which can be confirmed through slow query logs.
Cleanup is triggered when a large number of keys expire at the same time.
Short-term high-concurrency requests.
Sudden Increase in Connections
When the service is released, a large number of clients establish connections at the same time.
Improper connection pool configuration causes clients to create new connections frequently.
Network jitter causes old connections to be re-established after they are disconnected.
Sudden Drop in QPS
Abnormal network conditions on the client side or service interruption.
Traffic control is triggered on the instance (when bandwidth or the number of connections exceeds the limit).
The service is briefly unavailable during a primary/secondary failover.
Troubleshooting Methods
1. Determine the exact time period when the glitch occurred. We recommend viewing the monitor at 5-second granularity.
2. Compare the slow query logs, event alarms, and Ops operation records for that period.
3. Check whether the client logs contain corresponding abnormal error messages.
4. If glitches occur periodically, check whether any scheduled tasks (such as full scans or data synchronization) are triggered.