Backport kucero to caasp4 (#1025)

* Doc kucero signs kubelet server CSR

Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>

* Elaborates the worker node certificate signing flow

Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>

adoc/admin-security-certificates.adoc

@@ -97,9 +97,9 @@ The control plane certificates stored in the {kube} cluster on control plane nod
 |/var/lib/kubeket/pki/kubelet-client-current.pem
 |Client
 
-|<nodeName>
+|system:node:<nodeName>
 |kubelet-ca
-|/var/lib/kubeket/pki/kubelet.ca
+|/var/lib/kubelet/pki/kubelet-server-current.pem
 |Server
 
 |front-proxy-ca
@@ -485,36 +485,52 @@ kubectl rollout restart deployment/oidc-gangway -n kube-system
 
 == Automatic Certificate Renewal
 
-{productname} renews all certificates excluding `oidc-dex` and `oidc-gangway` automatically during the control plane update, see <<handling_updates>>.
+{productname} renews the control plane certificates and kubeconfigs automatically in two ways:
+
+. *During node upgrade*:
+when the node is upgraded, all the kubeadm managed certificates and kubeconfigs get rotated. _Note that, during node upgrade, neither the kubelet client certificate nor server certificate get rotated._ The time to rotate the kubelet client and server certificate is controlled by kubelet daemon.
+
+. *Via the `kucero` addon*:
+if the administrator is not willing to upgrade the cluster, the `kucero` (KUbernetes control plane CErtificate ROtation) addon rotates all the kubeadm managed certificates and kubeconfigs and signs kubelet server CSR. The kucero is a kubeadm checker/renewer in the form of a DaemonSet. It's job is to periodically check and renew control plane kubeadm managed certificates/kubeconfigs, and check the kubelet client and server enables auto rotation, and also a signer to sign kubelet server CSR.
 
-.kubelet client certificate
+.Time to rotate the kubelet client and server certificate
 [NOTE]
 ====
-During the kubelet client certificate signing flow, kubelet sends the kubelet client CSR into the {kube} cluster. The `kube-controller-manager` signs the kubelet client CSR with the Kubernetes CA cert/key pair. The `kucero` signs the kubelet server CSR with the kubelet CA cert/key pair.
+The kubelet client and server certificate renews automatically at approximately 70%-90% of the total lifetime of the certificate, the kubelet daemon would use new client and server certificates without downtime.
+====
 
-The kubelet client certificate renews automatically at approximately 70%-90% of the total lifetime of the certificate, the kubelet would use new client certificates without downtime.
+.Kubelet client and server certificate signing flow
+[NOTE]
+====
+The configuration which controls the kubelet daemon to send out the CSR within the {kube} cluster or not is controlled by the configuration `/var/lib/kubelet/config.yaml`. The key `rotateCertificates` controls the kubelet client certificate; the key `serverTLSBootstrap` controls the kubelet server certificate.
+
+When the client or server certificate is going to expire, the kubelet daemon sends the kubelet client or server CSR within the {kube} cluster. The `kube-controller-manager` signs the kubelet client CSR with the Kubernetes CA cert/key pair, the `kucero` signs the kubelet server CSR with the kubelet CA cert/key pair. Then, the kubelet daemon saves the signed certificate under the folder `/var/lib/kubelet/pki` and updates the client or server certificate symlink points to the latest signed certificate.
 
-The kubelet client certificate uses the `/var/lib/kubelet/pki/kubelet-client-current.pem`, this file is a symlink to the latest signed client certificate.
+The path of kubelet client certificate is `/var/lib/kubelet/pki/kubelet-client-current.pem`.
+The path of kubelet server certificate is `/var/lib/kubelet/pki/kubelet-server-current.pem`.
 ====
 
 === Control Plane Nodes Certificates Rotation
 
-When doing a control plane update, `skuba node upgrade apply` will run
-`kubeadm upgrade` commands behind the scenes. `kubeadm upgrade apply` and
-`kubeadm upgrade node` will renew and use new `kubeadm`
-managed certificates on the node, including those stored in kubeconfig files,
-regardless of the remaining time for which the certificate was still valid.
+Control Plane Node Certificates are rotated in two ways:
+
+. *During node upgrade*:
+when doing a control plane update, `skuba node upgrade apply` runs `kubeadm upgrade` commands behind the scenes. `kubeadm upgrade apply` and
+`kubeadm upgrade node` renews and uses new `kubeadm` managed certificates on the node, including those stored in kubeconfig files, regardless of the remaining time for which the certificate was still valid.
+
+. *Via the kucero addon:*
+.. kubeadm managed certificates/kubeconfigs:
+a kubeadm checker/renewer to periodical checks (default interval is 1 hour) the kubeadm managed certificates/kubeconfigs, and rotates the certificates/kubeconfigs if the residual time is less than the total time (default 720 hours). Administrators can change the default time to renew the certificates/kubeconfigs by adding `--renew-before=<duration>`` (duration format is XhYmZs) to the kucero daemonset _or_ change the default polling period for checking the certificates/kubeconfigs by adding `--polling-period=<duration>` (duration format is XhYmZs).
+.. kubelet client and server certificates:
+a kubelet configuration checker/updater to periodical checks (default interval is 1 hour) if the kubelet configuration enables the client and server auto rotation. If not, kucero will helps enable the client and server auto-rotation by configuring the `rotateCertificates: true` and `serverTLSBootstrap: true` in `/var/lib/kubelet/config.yaml`. After that, the kubelet daemon will send out the CSR within {kube} cluster if the client or server is going to expire, the corresponding CSR signer and approver will signs and approves the CSR, then the kubelet daemon saves the signed certificate under the folder `/var/lib/kubelet/pki` and updates the symlink points to the latest signed certificate.
 
-=== Worker Nodes Certificates Rotation
+=== Worker Node Certificate Rotation
 
-Running `skuba node upgrade apply` on a worker node applies the same kind
-of logic than the control plane nodes: it runs `kubeadm upgrade node`
-which only restarts the kubelet on worker nodes.
+Worker Node Certificates are rotated in one way:
 
-Kubelet configuration by default uses a `kubelet-client-current.pem` file, in
-its `pki` folder. This file is a symlink to the latest generated certificate.
-Restarting the kubelet effectively rotates the certificate to read the latest
-generated file.
+. *Via the kucero addon:*
+.. kubelet client and server certificates:
+a kubelet configuration checker/updater to periodical checks (default interval is 1 hour) if the kubelet configuration enables the client and server auto rotation. If not, kucero will helps enable the client and server auto-rotation by configuring the `rotateCertificates: true` and `serverTLSBootstrap: true` in `/var/lib/kubelet/config.yaml`. After that, the kubelet daemon will send out the CSR within {kube} cluster if the client or server is going to expire, the corresponding CSR signer and approver will signs and approves the CSR, then the kubelet daemon saves the signed certificate under the folder `/var/lib/kubelet/pki` and updates the symlink points to the latest signed certificate.
 
 == Manual Certificate Renewal