One place for hosting & domains

      Deploying Kubernetes on Bare Metal


      Container technology and its ability to run software when moving from one environment to another has helped it grow in popularity over virtual machines, due to its lighter weight and use of fewer resources. Kubernetes is one of these container technologies that, when paired with an optimal deployment model, has the capability to automate application orchestration, scaling and management across computing environments.

      In this piece, we will explore the process of deploying Kubernetes on bare metal servers.

      What is Kubernetes?

      Kubernetes is a portable, extensible platform for managing containerized workloads and services. It is an increasingly popular tool for moving workloads efficiently across platforms with less overhead than traditional virtual machines (VMs).

      Kubernetes’s capability to automate application orchestration, scaling and management makes it a very attractive lightweight platform for deploying workloads at large scale. It has been gaining momentum as a form of building cloud-like environments for web-scale type of applications, especially on bare metal servers.

      Why Deploy Kubernetes on Bare Metal?

      Kubernetes can be deployed as a service offered by cloud provider or on top of a VM layer or on dedicated bare metal servers—either on-premises or in a hosted environment. Each model has its advantages and drawbacks in terms of speed of availability, complexity, management overhead, cost and operational involvement.

      Bare metal offers the operational flexibility and scalability inherent in a cloud model in addition to on-demand capacity management. Bare metal servers allow for optimized configuration with better performance, efficient resource usage and predictable improved cost.

      Setting Up Kubernetes on a Bare Metal Server: A Step-By-Step Process

      Designing a High Availability Kubernetes Cluster on Bare Metal

      For this exercise, we’ll deploy Kubernetes on a cluster of two control nodes and two worker nodes with a standalone HAProxy load balancer (LB) in front of the control nodes. Each node is a bare metal server. Three control nodes are typically recommended for a Kubernetes cluster allowing control nodes to form quorum and agree on cluster state, but in this exercise, we’ll use two nodes to demonstrate the high availability setup. We will use the following configuration for this exercise:

      • Kubernetes 1.18.6 is being used on bare-metal machines with Ubuntu 18.04.
      • The control nodes are used for control and management of the cluster, and they run cluster-wide components such as apiserver, controller-manager, scheduler and etcd. Etcd is a distributed key-value store for the critical data of a distributed system. Within Kubernetes etcd is used as the backend for service discovery and stores the cluster’s state and configuration. Etcd can be deployed on its own standalone nodes but for this exercise we will keep etcd within the control nodes.
      • A standalone node is used for HAProxy to load balance traffic to the control nodes and support control plan redundancy.
      • The servers use a private network (or vlan) to communicate within the cluster. In our case the network is 172.31.113.0/26

      Jump to Step

      Step 1: Configuring HAProxy as a Load Balancer
      Step 2: Install Docker and Related Packages on All Kubernetes Nodes
      Step 3: Prepare Kubernetes on All Nodes
      Step 4: Install the Key-Value Store “Etcd” on the First Control Node and Bring Up the Cluster
      Step 5: Add a Second Control Node (master2) to the Cluster
      Step 6: Add the Worker Nodes

      Step 1: Configuring HAProxy as a Load Balancer

      In this step wewill configure HAProxy as a load balancer on a standalone server. This node could be a bare metal server or a cloud instance. To keep the deployment steps generic and easy to follow we will define the IP addresses of the nodes as environment variables on all the server nodes.

      #create a file with the
      list of IP addresses in the cluster
      cat > kubeiprc <<EOF
      export KBLB_ADDR=172.31.113.50
      export CTRL1_ADDR=172.31.113.38
      export CTRL2_ADDR=172.31.113.40
      export WORKER1_ADDR=172.31.113.45
      export WORKER2_ADDR=172.31.113.46
      EOF

      #add the IP addresses of the nodes as environment variables
      chmod +x kubeiprc
      source kubeiprc

       

      Install the haproxy package:


      apt-get update && apt-get upgrade && apt-get install -y haproxy
       

       

      Update the HAproxy config file “/etc/haproxyhaproxy.cfg” as follows:

      ##backup the current file
      mv /etc/haproxy/haproxy.cfg{,.back}

      ## Edit the file
      cat > /etc/haproxy/haproxy.cfg << EOF
      global
           user haproxy
           group haproxy
      defaults
           mode http
           log global
           retries 2
           timeout connect 3000ms
           timeout server 5000ms
           timeout client 5000ms
      frontend kubernetes
           bind $KBLB_ADDR:6443
           option tcplog
           mode tcp
           default_backend kubernetes-master-nodes
      backend kubernetes-master-nodes
           mode tcp
           balance roundrobin
           option tcp-check
           server k8s-master-0 $CTRL1_ADDR:6443 check fall 3 rise 2
           server k8s-master-1 $CTRL2_ADDR:6443 check fall 3 rise 2
      EOF

       

      To allow for failover, the HAProxy load balancer needs the ability to bind to an IP address that is nonlocal, meaning an address not assigned to a device on the local system. For that add the following configuration line to the sysctl.conf file:

      cat >> /etc/sysctl.conf <<EOF
      net.ipv4.ip_nonlocal_bind = 1
      EOF

       

      Update the system parameters and start or restart HAProxy:

      sysctl -p
      systemctl start haproxy
      systemctl restart haproxy 

       

      Check that the HAProxy is working:

      nc -v $KBLB_ADDR 6443

      Step 2: Install Docker and Related Packages on All Kubernetes Nodes

      Docker is used with Kubernetes as a container runtime to access images and execute applications within
      containers. We start by installing Docker on each node.

      First, define the environment variables for the nodes IP addresses as described in step 1:

      cat > kubeiprc <<EOF
      export KBLB_ADDR=172.31.113.50
      export CTRL1_ADDR=172.31.113.38
      export CTRL2_ADDR=172.31.113.40
      export WORKER1_ADDR=172.31.113.45
      export WORKER2_ADDR=172.31.113.42
      EOF

      #add the IP addresses of the nodes as environment variables
      chmod +x kubeiprc
      source kubeiprc

       

      Update the apt package index and install packages to allow apt to use a repository over HTTPS:

      apt-get update
      apt-get -y install
      apt-transport-https
           ca-certificates
           curl
           gnupg2
           software-properties-common

       

      Add Docker’s official GPG key:


      curl –fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add –

      Add docker to APT repository:


      add-apt-repository
      “deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable”
       

       

      Install Docker CE:


      apt-get update && apt-get install docker-ce

       

      Next confirm Docker is running:

      Step 3: Prepare Kubernetes on All Nodes, Both Control and Worker Ones

      Install the Kubernetes components on the node. This step applies to both control and worker nodes.

      Install kubelet, kubeadm, kubectl packages from the Kubernetes repository. Kubelet is the component that runs on all the machines in the cluster and is needed to start pods and containers. Kubeadm is the program that bootstraps the cluster. Kubectl is the command line utility to issue commands to the cluster.

      Add the Google repository key to ensure software authenticity:


      curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add –

      Add the Google repository to the default set:


      cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
      deb https://apt.kubernetes.io/ kubernetes-xenial main
      EOF

       

      Update and install the main packages:


      apt-get update && apt-get install -y kubelet kubeadm kubectl
       

       

      To ensure the version of these packages matches the Kubernetes control plan to be installed and avoid unexpected behavior, we need to hold back packages and prevent them from being automatically updated:


      apt-mark hold kubelet kubeadm kubectl

       

      After the package install, disable swap, as Kubernetes is not designed to use SWAP memory:


      swapoff -a
      sed –i ‘/ swap / s/^/#/’ /etc/fstab

       

      Verify kubeadm utility is installed:


      kubeadm
      version
       

      Change the settings such that both docker and kubelet use systemd as the “cgroup” driver. Create the file /etc/docker/daemon.json as follows:

      Repeat Steps 2 and 3 above for all Kubernetes nodes, both control and worker nodes.

      Step 4: Install the Key-Value Store “Etcd” on the First Control Node and Bring Up the Cluster

      It is now time to bring up the cluster, starting with the first control node at this step. On the first Control node, create a configuration file called kubeadm-config.yaml:

      cd /root

      cat > kubeadm-config.yaml << EOF
      apiVersion: kubeadm.k8s.io/v1beta1
      kind: ClusterConfiguration
      kubernetesVersion: stable
      apiServer:
          certSANs:
           – “$KBLB_ADDR”
      controlPlaneEndpoint: “$KBLB_ADDR:6443”
      etcd:
           local:
             endpoints:
             – https://$CTRL1_ADDR:2379
             – https://$CTRL2_ADDR:2379
             caFile: /etc/kubernetes/pki/etcd/ca.crt
             certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
             keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
      EOF
        

       

      Initialize the Kubernetes control plane:


      sudo kubeadm init –config=kubeadm-config.yaml –upload-certs
       

       

      If all configuration goes well, a message similar to the one below is displayed.

      Kubeadm commands are displayed for adding a control-plane node or worker nodes to the cluster. Save the last part of the output in a file as it would be needed later for adding nodes.

      Please note that the certificate-key gives access to cluster sensitive data. As a safeguard, uploaded-certs will be deleted in two hours.

      On the configured control node, apply the Weave CNI (Container Network Interface) network plugin. This plugin creates a virtual network that connects docker containers deployed across multiple hosts as if the containers are linked to a big switch. This allows containerized applications to easily communicate with each other.


      kubectl –kubeconfig /etc/kubernetes/admin.conf apply -f
           “https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d ‘n’)”

      Check that the pods of system components are running:


      kubectl –kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -w
      kubectl –kubeconfig /etc/kubernetes/admin.conf get nodes
       

      Step 5: Add a Second Control Node (master2) to the Cluster

      As the fist control node is running, we can now add a second control node using the “kubeadm join” command. To join the master2 node to the cluster, use the join command shown in the output at the initialization of the cluster control plane on control node 1.

      Add the new control node to the cluster. Make sure to copy the join command from your actual output in the first control node initialization instead of the output example below:


      kubeadm join 172.31.113.50:6443 –token whp74t.cslbmfbgh34ok21a
           –discovery-token-ca-cert-hash
      sha256:50d6b211323316fcb426c87ed8b604eccef0c1eff98d3a44f4febe13070488d2
           –control-plane –certificate-key
      2ee6f911cba3bddf19159e7b43d52ed446367cb903dc9e5d9e7969abccb70a4b
       

       

      The output confirm that the new node has joined the cluster.

      The “certificate-key” flag of the “kubeadm join“ command causes the control plane certificates to be downloaded from the cluster “kubeadm-certs” store to generate the key files locally. The security certificate expires and is deleted every two hours by default. In case of error, the certificate can be reloaded from the initial control node in the cluster with the command:

      sudo kubeadm init phase upload-certs –upload-certs

      Once the certificates are reloaded, they get updated and the “kubeadm join” needs to be modified to use the right “certificate-key” parameter.

      Check that nodes and pods on all control nodes are okay:


      kubectl
      kubeconfig /etc/kubernetes/admin.conf get nodes
      kubectl –kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system -w

      Step 6: Add the Worker Nodes

      Now worker nodes can be added to the Kubernetes cluster.

      To add worker nodes, run the “kubeadm join” command in the worker node as shown in the output at the initialization of the cluster control plane on the first control node (example below, use the appropriate command saved from the controller build):

      kubeadm join 172.31.113.50:6443 –token whp74t.cslbmfbgh34ok21a
           –discovery-token-ca-cert-hash
      sha256:50d6b211323316fcb426c87ed8b604eccef0c1eff98d3a44f4febe13070488d2

       

      The output confirms the worker node is added.

      Check the nodes that are in the cluster to confirm the addition:


      kubectl –kubeconfig /etc/kubernetes/admin.conf get nodes
       

      With this config we have deployed an HA Kubernetes cluster with two control nodes, two worker nodes and an HAProxy load balancer in front of the two control nodes.

      The cluster deployed can be extended in a similar way as described above with more control nodes or additional worker nodes. Another HAProxy load balancer can be added for high availability.

      Conclusion

      The Kubernetes deployment demonstrated above used a cluster of bare metal servers. Using INAP Bare Metal, the same deployment can span multiple sites using additional networking plugins to achieve a highly available and scalable distributed environment. INAP Performance IP® can further optimize the inter-site connectivity by providing low latency for better overall performance of Kubernetes clusters.

      Explore INAP Bare Metal.

      LEARN MORE

      Layachi Khodja


      READ MORE



      Source link


      Leave a Comment