etcd reading member-k0-key.pem every second (again)

[kaleal]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

I have observed that etcd is reading the private key from disk continuously. The issue observed when an audit rule has been applied to catch the private keys being read.
The issue was discovered running a kubernetes cluster with three nodes deployed with kubespray in three virtual machines managed by vagrant.

What did you expect to happen?

etcd should read the key at start up and keep it in memory for further use.

My previous ticket was closed without any chance of justifying my point of view.

Here goes the replica:
Supporting dynamically loading the certificate does not imply on reading it from disk at every call.
The certificate can be held for a specific amount of time, and replaced when that time is expired. That could be even configurable.

Reading the file at every call can cause a bottleneck and harm the performance.

How can we reproduce it (as minimally and precisely as possible)?

On a k8s control plane node install auditd.

apt-get install auditd

Create an audit rules for monitoring the key file

for key in $(ls /etc/ssl/etcd/ssl/*key.pem); do
  echo -a always,exit -F arch=b64 -F path=${key} -F perm=rwa -k etcd -k etcd-keys
done >/etc/audit/rules.d/etcd-keys.rules
systemctl restart auditd

Inspect audit logs

ausearch -k etcd-keys -i

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.16
Git SHA: f20bbad
Go Version: go1.22.7
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.5.16
API version: 3.5

Etcd configuration (command line flags or environment variables)

root@k0:~# cat /etc/etcd.env 
# Environment file for etcd v3.5.16
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.121.89:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.121.89:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_METRICS=basic
ETCD_LISTEN_CLIENT_URLS=https://192.168.121.89:2379,https://127.0.0.1:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://192.168.121.89:2380
ETCD_NAME=etcd1
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.121.89:2380,etcd2=https://192.168.121.79:2380,etcd3=https://192.168.121.32:2380
ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_SNAPSHOT_COUNT=10000
# Flannel need etcd v2 API
ETCD_ENABLE_V2=true

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-k0.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-k0-key.pem
ETCD_CLIENT_CERT_AUTH=true

ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-k0.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-k0-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=True



# CLI settings
ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/ssl/etcd/ssl/ca.pem
ETCDCTL_KEY=/etc/ssl/etcd/ssl/admin-k0-key.pem
ETCDCTL_CERT=/etc/ssl/etcd/ssl/admin-k0.pem

# ETCD 3.5.x issue
# https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ?utm_medium=email&utm_source=footer
ETCD_EXPERIMENTAL_INITIAL_CORRUPT_CHECK=True


ETCD_EXPERIMENTAL_WATCH_PROGRESS_NOTIFY_INTERVAL=5s

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+-------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+-------+-----------------------------+-----------------------------+------------+
| 4a694b80a6468586 | started | etcd2 | https://192.168.121.79:2380 | https://192.168.121.79:2379 |      false |
| 6a2e077545146330 | started | etcd1 | https://192.168.121.89:2380 | https://192.168.121.89:2379 |      false |
| a0ffe7ca1e9739ea | started | etcd3 | https://192.168.121.32:2380 | https://192.168.121.32:2379 |      false |
+------------------+---------+-------+-----------------------------+-----------------------------+------------+


$ etcdctl --endpoints=<member list> endpoint status -w table
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.121.79:2379 | 4a694b80a6468586 |  3.5.16 |  7.8 MB |     false |      false |         8 |      60238 |              60238 |        |
| https://192.168.121.89:2379 | 6a2e077545146330 |  3.5.16 |  7.7 MB |      true |      false |         8 |      60238 |              60238 |        |
| https://192.168.121.32:2379 | a0ffe7ca1e9739ea |  3.5.16 |  7.7 MB |     false |      false |         8 |      60238 |              60238 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

----
type=PROCTITLE msg=audit(02/26/25 16:58:57.139:6265) : proctitle=/usr/local/bin/etcd 
type=PATH msg=audit(02/26/25 16:58:57.139:6265) : item=0 name=/etc/ssl/etcd/ssl/member-k0-key.pem inode=1966155 dev=fe:01 mode=file,700 ouid=etcd ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(02/26/25 16:58:57.139:6265) : cwd=/ 
type=SYSCALL msg=audit(02/26/25 16:58:57.139:6265) : arch=x86_64 syscall=openat success=yes exit=207 a0=AT_FDCWD a1=0xc001efdbc0 a2=O_RDONLY|O_CLOEXEC a3=0x0 items=1 ppid=1 pid=854 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=etcd exe=/usr/local/bin/etcd subj=unconfined key=etcd key=etcd-keys 
----
type=PROCTITLE msg=audit(02/26/25 16:58:58.159:6266) : proctitle=/usr/local/bin/etcd 
type=PATH msg=audit(02/26/25 16:58:58.159:6266) : item=0 name=/etc/ssl/etcd/ssl/member-k0-key.pem inode=1966155 dev=fe:01 mode=file,700 ouid=etcd ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(02/26/25 16:58:58.159:6266) : cwd=/ 
type=SYSCALL msg=audit(02/26/25 16:58:58.159:6266) : arch=x86_64 syscall=openat success=yes exit=207 a0=AT_FDCWD a1=0xc003b7d650 a2=O_RDONLY|O_CLOEXEC a3=0x0 items=1 ppid=1 pid=854 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=etcd exe=/usr/local/bin/etcd subj=unconfined key=etcd key=etcd-keys 
----
type=PROCTITLE msg=audit(02/26/25 16:58:58.303:6267) : proctitle=/usr/local/bin/etcd 
type=PATH msg=audit(02/26/25 16:58:58.303:6267) : item=0 name=/etc/ssl/etcd/ssl/member-k0-key.pem inode=1966155 dev=fe:01 mode=file,700 ouid=etcd ogid=root rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(02/26/25 16:58:58.303:6267) : cwd=/ 
type=SYSCALL msg=audit(02/26/25 16:58:58.303:6267) : arch=x86_64 syscall=openat success=yes exit=207 a0=AT_FDCWD a1=0xc003b7dad0 a2=O_RDONLY|O_CLOEXEC a3=0x0 items=1 ppid=1 pid=854 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=etcd exe=/usr/local/bin/etcd subj=unconfined key=etcd key=etcd-keys

[ahrtr]

You raised this as a bug, what's the exact issue that impacts your business?

You also mentions the performance impact. If it's only related to performance, this ticket should be a feature request, also it'd be better to demo how much performance may be affected with or without the dynamically loading certificate feature.