Using NodeLocal DNS Cache in a TKE Cluster

Last updated: 2020-02-27 14:22:36

    Scenarios

    By running NodeLocal DNSCache in a form of Daemonset on the cluster node, you can greatly improve the DNS resolution performance in the cluster, and can effectively avoid DNS 5-second delay due to conntrack conflicts.

    This document describes in detail how to use NodeLocal DNS Cache in a TKE cluster.

    Principle

    A hostNetwork Pod is deployed on every node of a cluster by using DaemonSet. This Pod is node-cache, and can cache DNS requests for Pods on this node. If there are cache misses, this Pod will obtain them through a TCP request to the upstream kube-dns service. The principle is shown in the following figure:

    NodeLocal DNS Cache does not have high availability (HA), so there will be a single point of nodelocal dns cache failure (Pod Evicted/OOMKilled/ConfigMpa error/DaemonSet Upgrade). However, this issue is actually a common failure problem that will exist in any single point proxy (such as kube-proxy and cni pod).

    Prerequisites

    You have created a cluster of Kubernetes version 1.14 or above in the TKE Console, and nodes exist in this cluster.

    Directions

    1. Log in to the node. In the Linux terminal, execute the following command to obtain the CLUSTER IP of Kube-dns-upstream.
      kubectl -n kube-system get services kube-dns -o jsonpath="{.spec.clusterIP}"
      The returned result is as shown in the following figure. Record the CLUSTER IP.
    2. Deploy NodeLocal DNS Cache with one click. The YAML example is as follows:
      • The ConfigMap resource created in this step is the configuration file of coredns, where the UPSTREAM_CLUSTER_IP field must be replaced with the CLUSTER IP obtained in Step 1.
      • The DaemonSet resource created in this step is used to deploy the Local DNS Cache component.
    apiVersion: v1
    kind: ConfigMap
    metadata:
         name: node-local-dns
         namespace: kube-system
         labels:
           addonmanager.kubernetes.io/mode: Reconcile
    data:
         Corefile: |
           cluster.local:53 {
               errors
               cache {
                       success 9984 30
                       denial 9984 5
               }
               reload
               loop
               bind 169.254.20.10
               forward . ${UPSTREAM_CLUSTER_IP} {
                       force_tcp
               }
               prometheus :9253
               health 169.254.20.10:8080
               }
           in-addr.arpa:53 {
               errors
               cache 30
               reload
               loop
               bind 169.254.20.10
               forward . ${UPSTREAM_CLUSTER_IP} {
                       force_tcp
               }
               prometheus :9253
               }
           ip6.arpa:53 {
               errors
               cache 30
               reload
               loop
               bind 169.254.20.10
               forward . ${UPSTREAM_CLUSTER_IP} {
                       force_tcp
               }
               prometheus :9253
               }
           .:53 {
               errors
               cache 30
               reload
               loop
               bind 169.254.20.10
               forward . /etc/resolv.conf {
                       force_tcp
               }
               prometheus :9253
               }
        ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
         name: node-local-dns
         namespace: kube-system
         labels:
           k8s-app: node-local-dns
           kubernetes.io/cluster-service: "true"
           addonmanager.kubernetes.io/mode: Reconcile
    spec:
         updateStrategy:
           rollingUpdate:
             maxUnavailable: 10%
         selector:
           matchLabels:
             k8s-app: node-local-dns
         template:
           metadata:
             labels:
               k8s-app: node-local-dns
           spec:
             priorityClassName: system-node-critical
             serviceAccountName: node-local-dns
             hostNetwork: true
             dnsPolicy: Default  # Don't use cluster DNS.
             tolerations:
             - key: "CriticalAddonsOnly"
               operator: "Exists"
             containers:
             - name: node-cache
               image: k8s.gcr.io/k8s-dns-node-cache:1.15.7
               resources:
                 requests:
                   cpu: 25m
                   memory: 5Mi
               args: [ "-localip", "169.254.20.10", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream" ]
               securityContext:
                 privileged: true
               ports:
               - containerPort: 53
                 name: dns
                 protocol: UDP
               - containerPort: 53
                 name: dns-tcp
                 protocol: TCP
               - containerPort: 9253
                 name: metrics
                 protocol: TCP
               livenessProbe:
                 httpGet:
                   host: 169.254.20.10
                   path: /health
                   port: 8080
                 initialDelaySeconds: 60
                 timeoutSeconds: 5
               volumeMounts:
               - mountPath: /run/xtables.lock
                 name: xtables-lock
                 readOnly: false
               - name: config-volume
                 mountPath: /etc/coredns
               - name: kube-dns-config
                 mountPath: /etc/kube-dns
             volumes:
             - name: xtables-lock
               hostPath:
                 path: /run/xtables.lock
                 type: FileOrCreate
             - name: kube-dns-config
               configMap:
                 name: kube-dns
                 optional: true
             - name: config-volume
               configMap:
                 name: node-local-dns
                 items:
                   - key: Corefile
                     path: Corefile.base
    1. Set the specified DNS resolution access address of kubelet to the local DNS cache created in Step 2. This document provides the following two configuration methods, which you can choose according to your actual situation:
      • Execute the following commands in order, to modify the kubelet launch parameters and restart it.
    sed -i '/CLUSTER_DNS/c\CLUSTER_DNS="--cluster-dns=169.254.20.10"' /etc/kubernetes/kubelet
    systemctl restart kubelet
    • Restart after configuring the dnsconfig of a single Pod as needed. The YAML core references are as follows:
      • You must set the nameserver to 169.254.20.10.
      • To ensure the internal domain name of the cluster can be resolved normally, you must configure searches.
      • Suitably reducing the ndots value is useful for accelerating the external domain name access of clusters.
      • When the Pod does not use the internal domain name of a cluster with multiple dots, it is recommended that you set the value to 2.
        dnsConfig:
        nameservers: ["169.254.20.10"]
        searches: 
          - default.svc.cluster.local
          - svc.cluster.local
          - cluster.local
        options:
          - name: ndots
            value: "2" 

    Configuration Verification

    This test cluster is a Kubernetes version 1.14 cluster. After the NodeLocal DNSCache component is deployed through the preceding steps, you can perform verification by referring to the following methods:

    1. Select a debug pod, and restart after adjusting kubelet parameters or configuring dnsConfig.
    2. Dig Internet domain name, try to capture packets on the coredns pod.
    3. If it shows that 169.254.20.10 is working normally, it means that the NodeLocal DNSCache component has been deployed successfully. This is shown in the following figure:

    Was this page helpful?

    Was this page helpful?

    • Not at all
    • Not very helpful
    • Somewhat helpful
    • Very helpful
    • Extremely helpful
    Send Feedback
    Help