What Is Loopback?
In a TKE cluster, accessing an internal Ingress in iptables or IPVS mode may result in a 4-second delay or connection failure. In IPVS mode, accessing an internal LoadBalancer Service within the cluster may result in complete connection failure or unstable connectivity.
Loopback Scenario Quick Reference
1. Quick Identification
Use the following table to determine whether connection access is affected by the loopback phenomenon:
|
Super nodes + private network CLB + CLB Service | No risk | No processing is required. |
Super nodes + private network CLB + CLB Ingress | No risk | No processing is required. |
iptables mode + private network CLB + CLB Service | No risk | No processing is required. |
Access a Service by Service name or ClusterIP. | No risk | No processing is required. |
Public network CLB + CLB Service / CLB Ingress | No risk | No processing is required. |
Regular nodes/native nodes + IPVS mode + private network CLB + CLB Service + access to LoadBalancer IP address | At risk | Processing required. |
Regular nodes/native nodes + private network CLB + CLB Ingress + access to LoadBalancer IP address | At risk | Processing required. |
Note:
Why is there no loopback risk when accessing by Service name within the cluster?
When you use a Service name (such as my-svc.default.svc.cluster.local) or a ClusterIP, traffic is directly forwarded to the target Pod within the cluster without passing through the CLB, so no loopback issue occurs.
2. Default Behavior of Different Node Types for CLB Service in IPVS Mode
|
Private network CLB | General Nodes | Not intercepted (traffic goes through CLB) | At risk | By default, the CLB VIP is not intercepted within the cluster. Traffic is forwarded through the CLB, and a loopback may occur when traffic is on the same node. |
Private network CLB | Native Nodes | Not intercepted (traffic goes through CLB) | At risk | The behavior is the same as that of general nodes. |
Private network CLB | Super Nodes | Intercepted (traffic closed-loop within the cluster) | No risk | By default, the CLB VIP is intercepted within the cluster. Traffic is directly forwarded to the target Pod without passing through the CLB. |
Public network CLB | - | - | No risk | - |
Note:
What is "CLB VIP interception"?
When a Pod accesses the CLB VIP, the cluster network component directly converts the VIP to the backend Pod IP address on the local node, and traffic is forwarded within the cluster without passing through the CLB. This avoids the issue of the CLB forwarding traffic back to the same node.
3. Layer 4 (Service) Quick Reference
|
Regular nodes/native nodes + private network CLB + IPVS | Passes through CLB | Loopback risk exists | |
Regular nodes/native nodes + private network CLB + IPVS + intra-cluster forwarding enabled | Closed loop within the cluster, without passing through the CLB | None | Resolved. |
Regular nodes/native nodes + private network CLB + IPVS + SNAT enabled on CLB | Passes through CLB | None | Resolved. |
Super Nodes | Closed loop within the cluster, without passing through the CLB | None | No processing is required. |
iptables mode | Closed loop within the cluster, without passing through the CLB | None | No processing is required. |
4. Layer 7 (Ingress) Quick Reference
|
Private network CLB + SNAT disabled on CLB | Passes through CLB | Loopback risk exists | |
Private network CLB + SNAT enabled on CLB | Passes through CLB | None | Resolved. |
Attention:
Layer 7 loopback can only be resolved by enabling SNAT on the CLB. The intra-cluster forwarding solution is not applicable to Layer 7 scenarios.
Solution
Solution 1: Forwarding Within the Cluster
Applicable scenarios: regular nodes/native nodes + private network CLB + IPVS + Layer 4 Service
Principle: By adding an annotation, the cluster network component intercepts access to the CLB VIP on the node and directly forwards traffic to the target Pod within the cluster without passing through the external CLB, fundamentally avoiding loopback.
Configuration Method:
apiVersion: v1
kind: Service
metadata:
name: my-service
annotations:
service.cloud.tencent.com/prevent-loopback: "true"
spec:
type: LoadBalancer
selector:
app: my-app
ports:
- port: 80
targetPort: 8080
Note:
After it is enabled, traffic no longer passes through the CLB, so the capabilities of the CLB cannot be used.
The health check IP address is automatically changed to the 100.64 network segment, and no additional configuration is required.
Existing long-lived connections are not affected. Existing connections continue to pass through the CLB, while new connections use intra-cluster forwarding.
Solution 2: Enabling SNAT on the CLB Layer 4 Listener for the Service
Applicable scenarios: regular nodes/native nodes + private network CLB + IPVS + Layer 4 Service + traffic that needs to pass through the CLB
Principle: After SNAT is enabled on the CLB, the CLB replaces the source IP address with its own IP address when forwarding requests to the backend. In this way, when the backend Pod sends a response, the destination address is the CLB IP address, so the response must pass through the CLB and will not loop back within the node.
Configuration Method:
1. To apply for the allowlist, submit a ticket to request activation: CLB Layer 4 SNAT capability
2. Configure SNAT:
Enable the SNAT feature of the CLB Layer 4 listener by configuring TkeServiceConfig spec.loadBalancer.l4Listeners.snatEnable: true. Note:
The real client IP address can be obtained through the Proxy Protocol over TCP.
The real source IP address cannot be obtained over UDP.
Comparison Between Solution 1 and Solution 2 (Layer 4 Loopback)
|
Intercept the CLB VIP on the node to keep traffic within the cluster. | CLB replaces the source IP address so that return packets must pass through CLB. |
Does not pass through | Passes through |
None | None |
Unavailable (traffic does not pass through CLB) | Available |
Can be obtained | TCP can be obtained through Proxy Protocol, while UDP cannot be obtained. |
Solution 3: Enabling CLB SNAT for Layer 7 Ingress
Applicable scenarios: private network CLB + Layer 7 Ingress
Principle: After SNAT is enabled on the CLB, the CLB replaces the source IP address with its own IP address when forwarding requests to the backend. In this way, when the backend Pod sends a response, the destination address is the CLB IP address, so the response must pass through the CLB and will not loop back within the node.
Configuration Method:
1. To apply for the allowlist, submit a ticket to request activation: CLB Layer 7 SNAT capability
Keep Alive capability
2. Configure SNAT:
When the ingress-controller version is v2.9.0 or later, enable the SNAT feature of the CLB Layer 7 listener by configuring TkeServiceConfig spec.loadBalancer.l7Listeners.snatEnable: true. When the ingress-controller version is earlier than v2.9.0, enable the SNAT feature of the CLB Layer 7 listener by configuring TkeServiceConfig spec.loadBalancer.l7Listeners.keepaliveEnable: 1. Note:
After it is enabled, you need to obtain the real source IP address from the X-Forwarded-For HTTP Header.
Make sure that node configurations allow access from the LoadBalancer IP address network segment.
FAQs
Q1: Why Is There No Loopback Issue with Services on Super Nodes?
By default, super nodes intercept the VIP of the CLB associated with a Service within the cluster and directly forward traffic to the target Pod without passing through the CLB, so no loopback issue occurs.
Q2: Why Is There No Loopback Issue with Services in iptables Mode?
In iptables mode, destination address translation is completed before routing. Traffic is then directly forwarded within the cluster to the target Pod without passing through the CLB, so no loopback issue occurs.
Q3: Does Adding the service.cloud.tencent.com/prevent-loopback: "true" annotation Affect Existing Services?
Existing long-lived connections: They are not affected and continue to pass through the CLB. This is because existing conntrack entries are matched first, so the original path is retained.
New connections: They are forwarded within the cluster instead of passing through the CLB. This is because new SYN packets undergo routing decisions, match the local table, and are intercepted by IPVS.