Multi-Master Node Deployment

Last updated: 2021-11-18 12:47:30

    Overview

    A system with a single master may be less stable, and in some online service scenarios, multiple masters are required to ensure fault tolerance. To implement fault tolerance, GooseFSRuntime provides 3 masters, which are elected using Raft, a highly consistent, decentralized, highly available distributed protocol that is widely used in engineering.

    The feature is introduced as follows.

    Prerequisites

    Before running the sample code provided in this document, install Fluid by referring to Installation and check that all the components used by Fluid are running properly.

    $ kubectl get pod -n fluid-system
    goosefsruntime-controller-5b64fdbbb-84pc6   1/1     Running   0          8h
    csi-nodeplugin-fluid-fwgjh                  2/2     Running   0          8h
    csi-nodeplugin-fluid-ll8bq                  2/2     Running   0          8h
    csi-nodeplugin-fluid-dhz7d                  2/2     Running   0          8h
    dataset-controller-5b7848dbbb-n44dj         1/1     Running   0          8h
    

    Normally, you shall see a pod named dataset-controller, a pod named goosefsruntime-controller, and multiple pods named csi-nodeplugin. The number of csi-nodeplugin pods depends on the number of nodes in your Kubernetes cluster.

    Setting Up an Environment

    $ mkdir <any-path>/co-locality
    $ cd <any-path>/co-locality
    

    Example

    Check all nodes in your cluster

    $ kubectl get nodes
    NAME                       STATUS   ROLES    AGE     VERSION
    192.168.1.145   Ready    <none>   7d14h   v1.18.4-tke.13
    192.168.1.146   Ready    <none>   7d14h   v1.18.4-tke.13
    192.168.1.147   Ready    <none>   7d14h   v1.18.4-tke.13
    

    Check the Dataset resource object to be created

    apiVersion: data.fluid.io/v1alpha1
    kind: Dataset
    metadata:
     name: hbase
    spec:
     mounts:
       - mountPoint: https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/stable/
         name: hbase
    
    Note:

    To facilitate testing, mountPoint is set to Web UFS in this example. If you want to mount COS, see Mounting COS (COSN) to GooseFS.

    Create the Dataset resource object

    $ kubectl create -f dataset.yaml
    dataset.data.fluid.io/hbase created
    

    Check the GooseFSRuntime resource object to be created

    apiVersion: data.fluid.io/v1alpha1
    kind: GooseFSRuntime
    metadata:
     name: hbase
    spec:
     replicas: 3
     tieredstore:
       levels:
         - mediumtype: HDD
           path: /mnt/disk1
           quota: 2G
           high: "0.8"
           low: "0.7"
     master:
       replicas: 3
    

    Specify spec.master.replicas=3 to enable the Raft 3-master mode. The parameter must be set to a positive odd number.

    Create the GooseFSRuntime resource object and check its status

    $ kubectl create -f runtime.yaml
    goosefsruntime.data.fluid.io/hbase created
    
    $ kubectl get pod
    NAME                          READY   STATUS    RESTARTS   AGE
    hbase-fuse-4v9mq     1/1     Running   0          84s
    hbase-fuse-5kjbj     1/1     Running   0          84s
    hbase-fuse-tp2q2     1/1     Running   0          84s
    hbase-master-0       1/1     Running   0          104s
    hbase-master-1       1/1     Running   0          102s
    hbase-master-2       1/1     Running   0          100s
    hbase-worker-cx8x7   1/1     Running   0          84s
    hbase-worker-fjsr6   1/1     Running   0          84s
    hbase-worker-fvpgc   1/1     Running   0          84s
    

    Check the status of the GooseFSRuntime object

    NAME     MASTER PHASE   WORKER PHASE   FUSE PHASE   AGE
    hbase   Ready           Ready            Ready     15m
    

    Check that all PHASE values are Ready.

    Check the Raft status (leader/follower)

    Log in to the pod of one master:

    $ kubectl exec -ti hbase-master-0 bash
    $ goosefs fs masterInfo
    

    You can see that the one node is leader, and the other two are follower:

    current leader master: hbase-master-0:26000
    All masters: [hbase-master-0:26000, hbase-master-1:26000, hbase-master-2:26000]