Redis Instance Connection Failure

Last updated: 2021-10-20 11:26:57

    Error Description

    • Symptom 1: failed to connect to or log in to a TencentDB for Redis instance from a CVM instance.
    • Symptom 2: failed to connect to or log in to a TencentDB for Redis instance from the database management center (DMC).

    Possible Reasons

    • Network issues.
    • Security group issues.
    • Password issues.
    • The maximum number of connections has been reached.
    • Memory or shards have been used up.
    • Public network access (if needed) failed.
    • A high-availability (HA) switch occurred, the database service became unavailable, a read-only replica switch occurred, or the read-only replica service became unavailable, etc.

    Solutions

    1. Run telnet to locate where the error occurred (in TencentDB for Redis or your business).
    2. Check whether the error was caused by password issues.
    3. Modify the allowed maximum number of connections.
    4. Check whether the error was caused by write failure due to used-up memory or shards.
    5. Connect to the TencentDB for Redis instance over the public network by iptable-based forwarding.
    6. Check whether any of the following occurred: HA switch, unavailable database service, read-only replica switch, or unavailable read-only replica service.

    Troubleshooting Procedure

    Running telnet to locate where the error occurred (in TencentDB for Redis or your business)

    Run telnet in the command line tool to narrow down the cause of the error:

    [root@VM-4-10-centos ~]# telnet 10.x.x.34 6379
    Trying 10.x.x.34...
    Connected to 10.x.x.34.
    Escape character is '^]'.
    

    As shown above, if the result indicates that the connection is successful, the TencentDB for Redis instance runs normally. Please troubleshoot your business:

    1. Troubleshoot the network

    To connect over the private network, the CVM and TencentDB instances must be under the same account and in the same VPC in the same region, or both in the classic network.

    • If the CVM instance is in a VPC, while the Redis instance in the classic network, we recommend that you switch the network type of the Redis instance from classic network to VPC. For more information, see Configuring Network.
    • If the Redis instance is in a VPC, while the CVM instance in the classic network, we recommend that you switch the network type of the CVM instance from classic network to VPC. For more information, see Switch to VPC.
    • If the CVM and TencentDB for Redis instances are in different VPCs in the same region, we recommend that you migrate the Redis instance to the VPC of the CVM instance. For more information, see Configuring Network.
    • If the CVM and TencentDB for Redis instances are in different VPCs in different regions, we recommend that you create a CCN between the two VPCs.
    • If the CVM and TencentDB for Redis instances are in different VPCs under different accounts, we recommend that you create a CCN between the two VPCs.

    2. Troubleshoot security groups

    The CVM instance cannot connect to the TencentDB for Redis instance if their security groups are incorrect.

    • Incorrect CVM security group configuration
      To use the CVM instance to access the Redis instance, you need to configure an outbound rule in the security group of the CVM instance. If the target of the outbound rule isn't "0.0.0.0/0" and the protocol port isn't "ALL", the IP and port of the Redis instance should be added to the rule.

      1. Go to the Security Group page in the CVM console and click the name of the CVM-bound security group to enter its details page.
      2. On the Outbound rule tab, click Add Rule.
        Set Type to Custom, Target to the IP/IP range of the Redis instance, and Policy to Allow.
    • Incorrect Redis security group configuration
      To use the CVM instance to access the Redis instance, you need to configure an inbound rule in the security group of the Redis instance. If the source of the inbound rule isn't "0.0.0.0/0" and the protocol port isn't "ALL", the IP and port of the CVM instance should be added to the rule.

      1. Go to the Security Group page in the CVM console and click the name of the Redis-bound security group to enter its details page.
      2. On the Inbound rule tab, click Add Rule.
        Note that you also need to open the IP and port of the Redis instance in the inbound rule.
        Set Type to Custom, Source to the IP/IP range of the CVM instance, and Policy to Allow.
        Note:

        • The Redis instance uses private network port 6379 by default and supports customizing its port. If the default port is changed, the new port should be opened in the inbound rule of the Redis security group.
        • If the default port 6379 of the Redis instance is used, it should be opened in the inbound rule of the Redis security group.

    Troubleshooting the password

    Run the info command. If the following information is displayed, the password of the TencentDB for Redis instance is correct.

    [root@SNG-Qcloud /data/home/rickyu]# redis-cli -h 10.x.x.34 -p 6379 -a password
    10.x.x.2:6379> info cpu
    # CPU
    used_cpu_sys:1623.176000
    used_cpu_user:4649.572000
    used_cpu_sys_children:0.000000
    used_cpu_user_children:0.000000
    

    If NOAUTH Authentication required. is displayed, the password is incorrect.

    10.0.4.31:6379> info memory
    NOAUTH Authentication required.
    10.0.4.31:6379> 
    

    Solutions

    Log in to the TencentDB for Redis console and click an instance ID in the instance list to enter the instance details page, where you can reset the password. For more information, see Managing Accounts.

    Adjusting the maximum number of connections

    1. Log in to the TencentDB for Redis console, click an instance ID in the instance list, and enter the instance details page.
    2. Click Adjust next to Max Connections in Network Info.
      Note:

      You can modify the maximum number of proxy connections in the console. To modify the maximum number of Redis connections, please submit a ticket.

    3. On the System Monitoring > Monitoring Metrics tab, select the Connection Utilization metric to view its value.
      Proxy monitoring data:

      Redis monitoring data:

    Checking whether the memory or shards are used up

    If you receive the following error message:

    "-READONLY You can't write against a read only slave.\r\n"
    

    Log in to the TencentDB for Redis console, click an instance ID in the instance list, and select the System Monitoring tab to view memory utilization.

    If memory is used up, writes will fail. Please expand capacity immediately or adopt the allkeys-lru or volatile-lru eviction policy.

    Note:

    Instance data may be lost if the allkeys-lru eviction policy is adopted. Please assess the impact before doing so.

    Connecting over the public network by iptable-based forwarding

    TencentDB for Redis does not support public network access for the time being. You can use a CVM instance with a public IP for port forwarding so as to access TencentDB for Redis over the public network. For more information, see Connecting to TencentDB for Redis Instances (over Public Network).

    Note:

    iptable-based forwarding may be unstable; therefore, you are not recommended to access instances over the public network in the production environment.

    Checking whether any of the following occurred: HA switch, unavailable database service, read-only replica switch, or unavailable read-only replica service

    If the connection becomes abnormal or a large number of access errors or slow queries are reported at a certain point in time, and you receive an event alarm from Cloud Monitor, an abnormal event occurs.

    Configure event alarms in the Cloud Monitor console:

    Appendix

    Viewing network type and VPC information

    To enable connection between CVM and TencentDB for Redis instances over the private network, they must be under the same account and in the same VPC in the same region, or both in the classic network.

    Note:

    • If the instance lists both show Classic Network or VPC, it means that the networks of the CVM and TencentDB for Redis instances are of the same type.
    • If the instance lists both show the same VPC (in the same region), it means that the CVM and TencentDB for Redis instances are in the same VPC.
    • View the network type/VPC of CVM: log in to the CVM console and view network information in the instance list.
    • View the network type/VPC of TencentDB for Redis: log in to the TencentDB for Redis console and view Network in the instance list.