kubeadm requires kubelet.conf to use specific username, although kubelet doesn't require it

[vrutkovs]

What happened?

See issue in kubernetes/kubernetes#89824. If a cluster is created via kubeadm and External CA mode is used, kubeadm init will fail if supplied kubelet.conf doesn't have a user matching node registration name, although official docs don't put restrictions on credential name there.

Stacktrace:

I0206 12:04:35.524109     128 kubeletfinalize.go:90] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0206 12:04:35.524478     128 loader.go:373] Config loaded from file:  /etc/kubernetes/kubelet.conf
the file "/etc/kubernetes/kubelet.conf" does not contain authentication for user "spoke-all-certs-control-plane"
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runKubeletFinalizeCertRotation
	cmd/kubeadm/app/cmd/phases/init/kubeletfinalize.go:119
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:259
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:446
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:232
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	cmd/kubeadm/app/cmd/init.go:111
github.com/spf13/cobra.(*Command).execute
	vendor/github.com/spf13/cobra/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
	vendor/github.com/spf13/cobra/command.go:1040
github.com/spf13/cobra.(*Command).Execute
	vendor/github.com/spf13/cobra/command.go:968
k8s.io/kubernetes/cmd/kubeadm/app.Run
	cmd/kubeadm/app/kubeadm.go:50

What did you expect to happen?

kubeadm finalize works with custom user in kubelet's kubeconfig

How can we reproduce it (as minimally and precisely as possible)?

• Follow https://kubernetes.io/docs/setup/best-practices/certificates to generate certificates and kubeconfigs.
  When kubelet.conf is generated use arbitrary name. e.g. default-auth as suggested in the docs:

  KUBECONFIG=/etc/kubernetes/kubelet.conf kubectl config set-cluster default-cluster --server=https://spoke-all-certs-control-plane:6443 --certificate-authority /path/to/ca.crt --embed-certs
  KUBECONFIG=/etc/kubernetes/kubelet.conf kubectl config set-credentials "default-auth" --client-key /path/to/key --client-certificate /path/to/cert.crt --embed-certs
  KUBECONFIG=/etc/kubernetes/kubelet.conf kubectl config set-context default-system --cluster default-cluster --user default-auth
  KUBECONFIG=/etc/kubernetes/kubelet.conf kubectl config use-context default-system

• Use these files during kubeadm init stage

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux spoke-all-certs-control-plane 6.7.3-200.fc39.x86_64 kubernetes/kubernetes#1 SMP PREEMPT_DYNAMIC Thu Feb  1 03:29:52 UTC 2024 x86_64 GNU/Linux

Install tools

kind (running kubeadm internally)

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vrutkovs]

/sig cluster-lifecycle

[neolit123]

/transfer kubeadm

[neolit123]

i think we can just remove the validation; this does seem like something we don't need to error on.
PRs welcome.

i'm +0 whether we should backport a fix for older releases for this one. for < 1.30 users can follow the recommended naming.

kubeadm does not follow the "credential name" guidance in the docs FWIW:
https://kubernetes.io/docs/setup/best-practices/certificates/#configure-certificates-for-user-accounts

$ sudo cat /etc/kubernetes/*.conf | grep users -A 1
users:
- name: kubernetes-admin
--
users:
- name: system:kube-controller-manager
--
users:
- name: system:node:some-node-name
--
users:
- name: system:kube-scheduler
--
users:
- name: kubernetes-super-admin

it follows the "system:..." naming which arguably makes more sense.

[neolit123]

i'm +0 whether we should backport a fix for older releases for this one. for < 1.30 users can follow the recommended naming.

technically, for external CA mode on a host that has /etc/kubernetes/pki/ca.key and ca.crt users can call kubeadm init phase kubeconfig all and they don't need to follow the manual steps in https://kubernetes.io/docs/setup/best-practices/certificates/#configure-certificates-for-user-accounts

i guess our https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#external-ca-mode section lacks details on how to use kubeadm phases to generate certs and kubeconfig out of band.

[vrutkovs]

technically, for external CA mode on a host that has /etc/kubernetes/pki/ca.key and ca.crt users can call kubeadm init phase kubeconfig all and they don't need to follow the manual step

Correct, if just the CA passed kubeadm will generate correct kubelet.conf with system:... name. However if a custom valid kubelet.conf is supplied (i.e. to use short-lived certs there instead of 1 year) kubeadm will consider it "corrupted" although it would be accepted by kubelet.

Perhaps instead of "does this kubeconfig has authinfo with expected name" kubeadm should check if its valid - for instance https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/clientcmd/api/helpers.go#L47 seems to be a function which may be useful for this purpose. WDYT?

[neolit123]

i think it would be sufficient to get the auth for the current context and update it.