[Kubernetes (K8S) Kubespray] Use Kubespray to add or remove control-plane,master node into the exist kubernetes (K8S) cluster

Posted on 2021-08-28 Edited on 2023-09-02 In Cloud Native , Kubernetes (K8S) , Kubespray Views: Word count in article: 4.6k Reading time ≈ 4 mins.

Add or remove control-plane,master node into the exist kubernetes (K8S) cluster

This still be the easiest with Kubespray.

Usages

First, check current all node status(2 nodes, 2 control-plane,master nodes):

# kubectl get nodes
NAME    STATUS   ROLES                  AGE     VERSION
node1   Ready    control-plane,master   3d16h   v1.21.4
node2   Ready    control-plane,master   3d16h   v1.21.4

Add a new control-plane,master node

Add the new control-plane,master into kubernetes (K8S) cluster.

1) Add a new node to the inventory

Edit inventory/mycluster/hosts.yaml file to add node3 host.

all:
  hosts:
    node1:
      ansible_host: 192.168.8.90
      ip: 192.168.8.90
      access_ip: 192.168.8.90
      ansible_user: root
    node2:
      ansible_host: 192.168.8.91
      ip: 192.168.8.91
      access_ip: 192.168.8.91
      ansible_user: root
    # add new node3
    node3:
      ansible_host: 192.168.8.92
      ip: 192.168.8.92
      access_ip: 192.168.8.92
      ansible_user: root
  children:
    kube_control_plane:
      hosts:
        node1:
        node2:
        # add new node3
        node3:
    kube_node:
      hosts:
        node1:
        node2:
        # add new node3
        node3:
    etcd:
      hosts:
        node1:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
    calico_rr:
      hosts: {}

2) Run cluster.yml

Run ansible-playbook command with cluster.yml and --limit=node3.

1	$ ansible-playbook -i inventory/mycluster/hosts.yaml cluster.yml

Append the new host to the inventory and run cluster.yml. You can NOT use scale.yml for that.

3) Restart Docker or kube-system/nginx-proxy

In all hosts, restart docker or reboot the host. Kubespray will update its static config, but it needs to be restarted in order to reload.

# systemctl restart docker

# Or reboot host
# reboot

4) Check current node status

check current all node status(3 nodes, 3 control-plane,master nodes):

# kubectl get nodes
NAME    STATUS   ROLES                  AGE     VERSION
node1   Ready    control-plane,master   3d16h   v1.21.4
node2   Ready    control-plane,master   3d16h   v1.21.4
node3   Ready    control-plane,master   5m16s   v1.21.4

Remove the control-plane,master node

Remove the control-plane,master node from kubernetes (K8S) cluster.

1) Run remove-node.yml

Run ansible-playbook command with remove-node.yml and --e=node=node3.

1	$ ansible-playbook -i inventory/mycluster/hosts.yaml remove-node.yml -e node=node3

With the old node still in the inventory, run remove-node.yml. You need to pass -e node=NODE_NAME to the playbook to limit the execution to the node being removed. If the node you want to remove is not online, you should add reset_nodes=false and allow_ungraceful_removal=true to your extra-vars.

2) Check current node status

Check current all node status(2 nodes, 2 control-plane,master nodes, without work node):

# kubectl get nodes
NAME    STATUS   ROLES                  AGE     VERSION
node1   Ready    control-plane,master   3d16h   v1.21.4
node2   Ready    control-plane,master   3d16h   v1.21.4

1) Remove the control-plane,master from the inventory

Edit inventory/mycluster/hosts.yaml file to remove node3 host.

all:
  hosts:
    node1:
      ansible_host: 192.168.8.90
      ip: 192.168.8.90
      access_ip: 192.168.8.90
      ansible_user: root
    node2:
      ansible_host: 192.168.8.91
      ip: 192.168.8.91
      access_ip: 192.168.8.91
      ansible_user: root
  children:
    kube_control_plane:
      hosts:
        node1:
        node2:
    kube_node:
      hosts:
        node1:
        node2:
    etcd:
      hosts:
        node1:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
    calico_rr:
      hosts: {}

FAQs

Check apiserver Certificate

1 2	# On all nodes, check apiserver Certificate. $ openssl x509 -text -noout -in /etc/kubernetes/ssl/apiserver.crt

Update apiserver Certificate

# On control-plane,master nodes
$ cd /etc/kubernetes

$ kubeadm init phase certs apiserver --config kubeadm-config.yaml

error downloading certs

error execution phase control-plane-prepare/download-certs: error downloading certs: error downloading the secret: Secret \"kubeadm-certs\" was not found in the \"kube-system\" Namespace. This Secret might have expired. Please, run `kubeadm init phase upload-certs --upload-certs` on a control plane to generate a new one

Manually upload Certificate.

1 2	# On control-plane,master nodes $ kubeadm init phase upload-certs --upload-certs

Run reset.yml

If you are doing this after you ended up with a broken master, be sure to run reset.yml using the parameter --limit=<broken_master_hostname> before continuing. If you take the precaution of recreating the certificate before adding the new master node, you won’t need this.

Run ansible-playbook command with reset.yml and --limit=node3.