MountVolume.SetUp failed for encrypted XFS volume #648
Labels
bug
Something isn't working
Comments
|
I could not reproduce your issue, could you maybe provide use the debug logs? See https://github.com/hetznercloud/csi-driver/blob/main/docs/debug-logs.md |
|
While
outputs nothing in debug mode
repeats these lines: |
|
We still cannot reproduce the error, we would need you to run a few command to gather some information. In the
cat /etc/os-release
cat /etc/mtab
apk add lsblk
lsblk --fs
cryptsetup luksDump /dev/disk/by-id/scsi-0HC_Volume_101069871
cryptsetup -v status scsi-0HC_Volume_101069871umount /dev/mapper/scsi-0HC_Volume_101069871
mount -v -t xfs /dev/mapper/scsi-0HC_Volume_101069871 /var/lib/kubelet/pods/860d1611-4be6-4fee-a7e2-960f337ddb75/volumes/kubernetes.io~csi/pvc-ef6bff65-56ea-4e6d-9e61-3050355f628b/mount |
|
I tested it again with a fresh deployment:
Result is reproducable: As you asked, I opened a shell:
Neither pod nor PVC are mentioned in I couldn't do these steps because cluster is behind a proxy: Finally, I tried to mount it manually: |
|
Sadly the mount command does not give us enough details, could you please share us the kernel logs after a mount failure? Same as above, after shelling in the hcloud-csi-node container, you could run: dmesgThe kernel logs from the node might be useful, if the logs from within the hcloud-csi-node container are not helpful enough. |
|
These lines are repeated dozens of times in |
|
This seems related to https://gitlab.alpinelinux.org/alpine/aports/-/issues/15744, could you maybe try the suggestions from this ticket, to see if it solves the problem for you? Also, if you could provide us information about the xfs partition, e.g. using Maybe running Sorry for all this back and forth, I sadly still cannot reproduce this. Thanks a lot for continuing to provide us debugging information! |
|
I suspect that the node's kernel must be updated to fix this, either by upgrading the kernel itself or by upgrade the worker to e.g. ubuntu-24.04. On our dev/test nodes, we currently have: |
|
After a recent upgrade we use the same Kernel version on our worker nodes (still on Ubuntu 22.04): I've tried to format the filesystem manually but failed every time: A bit unexpected: But
No worries. Keep up with your good work :) |
|
Did you run the commands above using the new csi-driver containers? If not, could you try that? Between csi-driver 2.7.1 and 2.8.0, the alpine image was bumped from alpine 3.15 to alpine 3.20 1e18db3 I noticed some differences between our 2 filesystem (note that the xfs_info was executed with csi-driver 2.7.1): Did you by any chance format the volume using custom flags? The same command using the csi-driver 2.8.0: The mkfs.xfs command might have failed because the flags were probably not available in csi-driver 2.7.1 (alpine 3.15). |
|
Container definitely runs on v2.8.0: Custom flags are not used. |
|
Hey, i am a workmate from @bheisig . Than i formatted the volume directly via the hcloud-csi-driver container. Now it was possible to mount the volume fully functional. Logs fixed: |
|
We finally managed to reproduce the bug. Setting The mismatch between the supported XFS options between the csi-driver container (alpine) and the node kernel is at the source of this bug. While the csi-driver formats the file system with the We will release a minimal bug fix that forces the use of We are also considering to only support a minimal kernel version (e.g. >=5.15) by default, using for example |
jooola
added a commit
to jooola/csi-driver
that referenced
this issue
Aug 8, 2024
By default, the csi-driver formats the file system with the nrext64=1 flag.
But the mismatch between the supported XFS options between the csi-driver container (alpine 3.20) and the node kernel (5.15 on ubuntu 22.04) prevent the xfs filesystem to be mounted with:
Superblock has unknown incompatible features (0x20) enabled
Closes hetznercloud#648
jooola
added a commit
that referenced
this issue
Aug 8, 2024
By default, the `hcloud-csi-driver:2.8.0` (`alpine 3.20`) formats the file system with the `nrext64=1` flag. But the mismatch between the supported XFS options between the csi-driver container and the node kernel (5.15 on ubuntu 22.04) prevent the XFS file system to be mounted with: ``` Superblock has unknown incompatible features (0x20) enabled ``` This change disable the `nrext64` flag when formatting new XFS file systems. Closes #648
jooola
pushed a commit
that referenced
this issue
Aug 8, 2024
🤖 I have created a release *beep* *boop* --- ## [2.9.0](v2.8.0...v2.9.0) (2024-08-08) ### Features * truncate poll exponential backoff function to max 10s ([#674](#674)) ([5785418](5785418)) ### Bug Fixes * xfs options compatibility between alpine and node kernel ([#689](#689)) ([13d853b](13d853b)), closes [#648](#648) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
lukasmetzner
pushed a commit
that referenced
this issue
Oct 10, 2024
By default, the `hcloud-csi-driver:2.8.0` (`alpine 3.20`) formats the file system with the `nrext64=1` flag. But the mismatch between the supported XFS options between the csi-driver container and the node kernel (5.15 on ubuntu 22.04) prevent the XFS file system to be mounted with: ``` Superblock has unknown incompatible features (0x20) enabled ``` This change disable the `nrext64` flag when formatting new XFS file systems. Closes #648
lukasmetzner
pushed a commit
that referenced
this issue
Oct 10, 2024
🤖 I have created a release *beep* *boop* --- ## [2.9.0](v2.8.0...v2.9.0) (2024-08-08) ### Features * truncate poll exponential backoff function to max 10s ([#674](#674)) ([5785418](5785418)) ### Bug Fixes * xfs options compatibility between alpine and node kernel ([#689](#689)) ([13d853b](13d853b)), closes [#648](#648) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TL;DR
Since version 2.8.0 k8s is unable to mount encrypted PVC with file system type XFS.
Expected behavior
hcloud volume will be attached to server, encrypted and mounted with file system type XFS
Observed behavior
kubectl -n <redacted> get events -w:Minimal working example
Log output
Journal on k8s worker node:
kubectl -n kube-system logs -f -l app.kubernetes.io/instance=hcloud-csiduring this time: nothingAdditional information
kubectl version:k8s worker node:
lsb_release -a:uname -a:The text was updated successfully, but these errors were encountered: