Tencent Kubernetes Engine (TKE) provides the default Global Router network mode, in which, routing can be implemented for container IP addresses (not overlapping with VPC IP ranges) within VPCs based on the global routing capabilities of VPCs. The principle of this mode is that the underlying VPC forwards traffic to the node of the corresponding pod CIDR based on the container IP address, and then the traffic is routed to the pod of the corresponding container IP address through network bridge cbr0, as shown in the figure below:
TKE also provides the VPC-CNI network mode. In this mode, an ENI is inserted into each pod, the IP range falls within the VPC IP range, and inter-pod communication is enabled through ENI routing, as shown in the figure below:
Compared with Global Router, the advantages and applicable scenarios of VPC-CNI are as follows:
|Without the need for a network bridge, the network forwarding performance is enhanced by about 10%||This mode is suitable for scenarios with high network latency requirements|
|Support static pod IP addresses||This mode is suitable for scenarios that rely on static container IP addresses, for example, migration from a traditional architecture to a container platform and security policy restrictions on IP addresses|
|Support direct connection between LBs and pods||This mode is suitable for scenarios where direct connection between LBs and pods is desired|
Log in to the TKE console.
TKE provides two methods to enable VPC-CNI:
Method 1: when creating a cluster, select the VPC-CNI network plug-in, as shown in the figure below:
When using Method 1 to enable VPC-CNI, choose Advanced Settings > Configure IP Repossession Policy.
Method 2: when creating a cluster, select the Global Router network plug-in. On the basic information page of the cluster, enable the VPC-CNI mode (by default, both modes are enabled), as shown in the figure below:
In static-IP scenarios, after enabling VPC-CNI, you need to configure the IP repossession policy, which specifies how long after pods are terminated their IP addresses are returned. Pods with non-static IP addresses are not affected by these settings because their IP addresses are immediately released upon pod termination, as shown in the figure below:
To use VPC-CNI, ensure that rp_filter is disabled. The following shows a sample code:
sysctl -w net.ipv4.conf.all.rp_filter=0 sysctl -w net.ipv4.conf.default.rp_filter=0
tke-cni-agent component automatically configures the node kernel parameters. If you manually configure the kernel parameters and enable rp_filter, network connection will fail.
tke.cloud.tencent.com/networks: "tke-route-eni", for pods and add requests and limits for one of the containers, such as
tke.cloud.tencent.com/eni-ip: "1". The following shows a sample code:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: annotations: tke.cloud.tencent.com/networks: "tke-route-eni" labels: app: nginx spec: containers: - name: nginx image: nginx resources: requests: tke.cloud.tencent.com/eni-ip: "1" limits: tke.cloud.tencent.com/eni-ip: "1"
The static IP feature is only applicable to StatefulSet-type workloads. You can enable static IP addresses by using the following two methods:
tke.cloud.tencent.com/vpc-ip-claim-delete-policy: Never, for the StatefulSet. The following shows a sample code:
apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: busybox spec: serviceName: "busybox" replicas: 1 template: metadata: annotations: tke.cloud.tencent.com/networks: tke-route-eni tke.cloud.tencent.com/vpc-ip-claim-delete-policy: Never labels: app: busybox spec: terminationGracePeriodSeconds: 0 containers: - name: busybox image: busybox command: ["sleep", "10000000000"] resources: requests: tke.cloud.tencent.com/eni-ip: "1" limits: tke.cloud.tencent.com/eni-ip: "1"
If you configure the IP repossession policy when enabling VPC-CNI, you can modify the policy later by manually modifying the launch parameter of the ipamd component, namely
kubectl -n kube-system edit deployments.v1.apps tke-eni-ipamd. The following shows a sample code:
spec: containers: - args: - --clusterid - cls-kjqul1ir - --claim-expired-duration - 10m0s
--claim-expired-durationto the specified value.
If the number of subnet IP addresses for VPC-CNI is insufficient, you can manually modify the configuration of the ipamd component to add subnets. To do this, edit configmap of
tke-eni-ipamd under the
kube-system namespace, namely
kubectl -n kube-system edit configmap tke-eni-ipamd. The following shows a sample code:
apiVersion: v1 kind: ConfigMap metadata: namespace: kube-system name: tke-eni-ipamd data: TKE_ENI_IPAMD_SUBNET_ID: subnet-4k3fdq3f:subnet-aa7clla5 TKE_ENI_IPAMD_VPC_ID: vpc-409o11tu
- Add the IDs of the subnets to be added to
TKE_ENI_IPAMD_SUBNET_IDand separate them with colons (
:). Note that the subnets to be added must be empty, that is, they do not contain any cloud resources such as CVMs and CLBs. Otherwise, a conflict will occur.
- If a configuration exists for TKE_ENI_IPAMD_ZONE, ignore it. This configuration has been discarded.
- After modifying the configuration of the ipamd component, you must run the following command to delete and rebuild the ipamd pod so that the modification can take effect:
kubectl -nkube-system get po -ocustom-columns=Name:.metadata.name | grep ipamd | kubectl -nkube-system delete po