[Kubernetes (K8S) Certificate] Use update-kube-cert shell script to extend Kubernetes (K8S) certificates expiration to 10 years

update-kube-cert

update-kube-cert is used to extended kubernetes cluster certificates that have expired or are about to expire.

This script is suitable for all k8s version cluster certificate update (using kubeadm initialized cluster).

The validity period of the certificate generated by kubeadm is 1 year, this script can update the validity period of the certificate generated by kubeadm to 10 years.

This script only processes the certificate on the master node. The kubelet certificate of the work node is automatically rotated and updated by default. You don’t need to care about the expiration issue, you only need to care about the certificate on the master node.

Usages

The script only needs to be executed on the master node, not on the work node

If there is no etcd related certificate, you only need to update the master certificate, see here (less than or equal to v1.9 version, etcd does not use TLS connection by default)

Follow the steps below to update the certificate by default

Download script

1
2
3
4
5
$ git clone https://github.com/yuyicai/update-kube-cert.git

$ cd update-kubeadm-cert

$ chmod 755 update-kubeadm-cert.sh

When executing, please use ./update-kubeadm-cert.sh all or bash update-kubeadm-cert.sh all, do not use sh update-kubeadm-cert.sh all, because some Linux distributions sh are not linked to bash. May be incompatible

Update certificate

If there are multiple master nodes, execute once on each master node

Excuting an order:

1
2
3
4
$ ./update-kubeadm-cert.sh all

# Not etcd
# $ ./update-kubeadm-cert.sh master

Output similar information

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
CERTIFICATE EXPIRES
/etc/kubernetes/controller-manager.config Sep 12 08:38:56 2022 GMT
/etc/kubernetes/scheduler.config Sep 12 08:38:56 2022 GMT
/etc/kubernetes/admin.config Sep 12 08:38:56 2022 GMT
/etc/kubernetes/pki/ca.crt Sep 11 08:38:53 2031 GMT
/etc/kubernetes/pki/apiserver.crt Sep 12 08:38:54 2022 GMT
/etc/kubernetes/pki/apiserver-kubelet-client.crt Sep 12 08:38:54 2022 GMT
/etc/kubernetes/pki/front-proxy-ca.crt Sep 11 08:38:54 2031 GMT
/etc/kubernetes/pki/front-proxy-client.crt Sep 12 08:38:54 2022 GMT
/etc/kubernetes/pki/etcd/ca.crt Sep 11 08:38:55 2031 GMT
/etc/kubernetes/pki/etcd/server.crt Sep 12 08:38:55 2022 GMT
/etc/kubernetes/pki/etcd/peer.crt Sep 12 08:38:55 2022 GMT
/etc/kubernetes/pki/etcd/healthcheck-client.crt Sep 12 08:38:55 2022 GMT
/etc/kubernetes/pki/apiserver-etcd-client.crt Sep 12 08:38:56 2022 GMT
[2021-09-12T16:41:25.93+0800][INFO] backup /etc/kubernetes to /etc/kubernetes.old-20210912
[2021-09-12T16:41:25.93+0800][INFO] updating...
[2021-09-12T16:41:25.99+0800][INFO] updated /etc/kubernetes/pki/etcd/server.conf
[2021-09-12T16:41:26.04+0800][INFO] updated /etc/kubernetes/pki/etcd/peer.conf
[2021-09-12T16:41:26.07+0800][INFO] updated /etc/kubernetes/pki/etcd/healthcheck-client.conf
[2021-09-12T16:41:26.11+0800][INFO] updated /etc/kubernetes/pki/apiserver-etcd-client.conf
[2021-09-12T16:41:26.54+0800][INFO] restarted etcd
[2021-09-12T16:41:26.60+0800][INFO] updated /etc/kubernetes/pki/apiserver.crt
[2021-09-12T16:41:26.64+0800][INFO] updated /etc/kubernetes/pki/apiserver-kubelet-client.crt
[2021-09-12T16:41:26.69+0800][INFO] updated /etc/kubernetes/controller-manager.conf
[2021-09-12T16:41:26.74+0800][INFO] updated /etc/kubernetes/scheduler.conf
[2021-09-12T16:41:26.79+0800][INFO] updated /etc/kubernetes/admin.conf
[2021-09-12T16:41:26.79+0800][INFO] backup /root/.kube/config to /root/.kube/config.old-20210912
[2021-09-12T16:41:26.80+0800][INFO] copy the admin.conf to /root/.kube/config
[2021-09-12T16:41:26.85+0800][INFO] updated /etc/kubernetes/kubelet.conf
[2021-09-12T16:41:26.88+0800][INFO] updated /etc/kubernetes/pki/front-proxy-client.crt
[2021-09-12T16:41:28.70+0800][INFO] restarted apiserver
[2021-09-12T16:41:29.17+0800][INFO] restarted controller-manager
[2021-09-12T16:41:30.07+0800][INFO] restarted scheduler
[2021-09-12T16:41:30.13+0800][INFO] restarted kubelet
[2021-09-12T16:41:30.14+0800][INFO] done!!!
CERTIFICATE EXPIRES
/etc/kubernetes/controller-manager.config Sep 11 08:41:26 2031 GMT
/etc/kubernetes/scheduler.config Sep 11 08:41:26 2031 GMT
/etc/kubernetes/admin.config Sep 11 08:41:26 2031 GMT
/etc/kubernetes/pki/ca.crt Sep 11 08:38:53 2031 GMT
/etc/kubernetes/pki/apiserver.crt Sep 11 08:41:26 2031 GMT
/etc/kubernetes/pki/apiserver-kubelet-client.crt Sep 11 08:41:26 2031 GMT
/etc/kubernetes/pki/front-proxy-ca.crt Sep 11 08:38:54 2031 GMT
/etc/kubernetes/pki/front-proxy-client.crt Sep 11 08:41:26 2031 GMT
/etc/kubernetes/pki/etcd/ca.crt Sep 11 08:38:55 2031 GMT
/etc/kubernetes/pki/etcd/server.crt Sep 11 08:41:25 2031 GMT
/etc/kubernetes/pki/etcd/peer.crt Sep 11 08:41:26 2031 GMT
/etc/kubernetes/pki/etcd/healthcheck-client.crt Sep 11 08:41:26 2031 GMT
/etc/kubernetes/pki/apiserver-etcd-client.crt Sep 11 08:41:26 2031 GMT

The following certificates and kubeconfig configuration files will be updated

1
2
3
4
5
6
7
8
9
10
11
12
13
14
/etc/kubernetes
├── admin.conf
├── controller-manager.conf
├── scheduler.conf
├── kubelet.conf
└── pki
├── apiserver.crt
├── apiserver-etcd-client.crt
├── apiserver-kubelet-client.crt
├── front-proxy-client.crt
└── etcd
├── healthcheck-client.crt
├── peer.crt
└── server.crt

See update-kube-cert/other.md at master · yuyicai/update-kube-cert - https://github.com/yuyicai/update-kube-cert/blob/master/other.md to learn more.

Rollback of certificate update failure

The script will automatically back up the /etc/kubernetes directory to the /etc/kubernetes.old-$(date +%Y%m%d) directory (example of naming the backup directory: kubernetes.old-20200325)

If the renewal of the certificate fails and needs to be rolled back, manually backup the /etc/kubernetes.old-$(date +%Y%m%d) directory over the /etc/kubernetes directory

Other

The following content has nothing to do with the script, just a few words

  • Versions greater than or equal to v1.15 can directly use kubeadm alpha certs renew <cert_name> to renew the certificate validity period. After the renewal, the certificate validity period will be extended by 1 year

  • Use kubeadm alpha certs renew <cert_name> if the version is less than v1.17

FAQs

Can’t open /etc/kubernetes/pki/etcd/ca.crt for reading, No such file or directory

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ ./update-kubeadm-cert.sh all
CERTIFICATE EXPIRES
/etc/kubernetes/controller-manager.config Oct 15 10:15:06 2022 GMT
/etc/kubernetes/scheduler.config Oct 15 10:15:07 2022 GMT
/etc/kubernetes/admin.config Oct 15 10:15:06 2022 GMT
/etc/kubernetes/pki/ca.crt Oct 13 10:15:04 2031 GMT
/etc/kubernetes/pki/apiserver.crt Oct 15 10:15:04 2022 GMT
/etc/kubernetes/pki/apiserver-kubelet-client.crt Oct 15 10:15:04 2022 GMT
/etc/kubernetes/pki/front-proxy-ca.crt Oct 13 10:15:04 2031 GMT
/etc/kubernetes/pki/front-proxy-client.crt Oct 15 10:15:05 2022 GMT
Can't open /etc/kubernetes/pki/etcd/ca.crt for reading, No such file or directory
139914442377024:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/etc/kubernetes/pki/etcd/ca.crt','r')
139914442377024:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
unable to load certificate
...

If etcd was installed independently by Kubenetes, just run update-kubeadm-cert.sh master to fix that issue.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
$ ./update-kubeadm-cert.sh master
CERTIFICATE EXPIRES
/etc/kubernetes/controller-manager.config Nov 24 09:08:12 2022 GMT
/etc/kubernetes/scheduler.config Nov 24 09:08:12 2022 GMT
/etc/kubernetes/admin.config Nov 24 09:08:11 2022 GMT
/etc/kubernetes/pki/ca.crt Oct 13 10:15:04 2031 GMT
/etc/kubernetes/pki/apiserver.crt Nov 24 09:08:11 2022 GMT
/etc/kubernetes/pki/apiserver-kubelet-client.crt Nov 24 09:08:12 2022 GMT
/etc/kubernetes/pki/front-proxy-ca.crt Oct 13 10:15:04 2031 GMT
/etc/kubernetes/pki/front-proxy-client.crt Nov 24 09:08:12 2022 GMT
[WARNING] does not backup, /etc/kubernetes.old-20211124 already exists
[INFO] updating...
[INFO] updated /etc/kubernetes/pki/apiserver.crt
[INFO] updated /etc/kubernetes/pki/apiserver-kubelet-client.crt
[INFO] updated /etc/kubernetes/controller-manager.conf
[INFO] updated /etc/kubernetes/scheduler.conf
[INFO] updated /etc/kubernetes/admin.conf
[INFO] backup /root/.kube/config to /root/.kube/config.old-20211124
[INFO] copy the admin.conf to /root/.kube/config
[INFO] does not need to update kubelet.conf
[INFO] updated /etc/kubernetes/pki/front-proxy-client.crt
./update-kubeadm-cert.sh: line 314: docker: command not found
[INFO] restarted apiserver
./update-kubeadm-cert.sh: line 314: docker: command not found
[INFO] restarted controller-manager
./update-kubeadm-cert.sh: line 314: docker: command not found
[INFO] restarted scheduler
[INFO] restarted kubelet
[INFO] done!!!
CERTIFICATE EXPIRES
/etc/kubernetes/controller-manager.config Nov 20 09:21:42 2031 GMT
/etc/kubernetes/scheduler.config Nov 20 09:21:42 2031 GMT
/etc/kubernetes/admin.config Nov 20 09:21:42 2031 GMT
/etc/kubernetes/pki/ca.crt Oct 13 10:15:04 2031 GMT
/etc/kubernetes/pki/apiserver.crt Nov 20 09:21:42 2031 GMT
/etc/kubernetes/pki/apiserver-kubelet-client.crt Nov 20 09:21:42 2031 GMT
/etc/kubernetes/pki/front-proxy-ca.crt Oct 13 10:15:04 2031 GMT
/etc/kubernetes/pki/front-proxy-client.crt Nov 20 09:21:43 2031 GMT

Then, check Kubernetes Certificates.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1124 09:31:07.969752 1655 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]

CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Nov 20, 2031 09:21 UTC 9y no
apiserver Nov 20, 2031 09:21 UTC 9y ca no
apiserver-kubelet-client Nov 20, 2031 09:21 UTC 9y ca no
controller-manager.conf Nov 20, 2031 09:21 UTC 9y no
front-proxy-client Nov 20, 2031 09:21 UTC 9y front-proxy-ca no
scheduler.conf Nov 20, 2031 09:21 UTC 9y no

CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 20, 2031 10:15 UTC 9y no
front-proxy-ca Nov 20, 2031 10:15 UTC 9y no

References

[1] yuyicai/update-kube-cert: K8s集群证书过期处理,更新kubeadm生成的证书有效期为10年。针对旧版集群(小于v1.15),当然大于等于v1.15也是可以用这个脚本更新,新版可直接kubeadm alpha certs renew <cert_name>更新 (deal with K8s cluster certificate expired) - https://github.com/yuyicai/update-kube-cert

[2] Certificate Management with kubeadm | Kubernetes - https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/