Delete pvc sometimes may not delete volume in heketi

[yangyuw]

As far as i know, delete pvc will clean the volume in heketi, but I found some times delete pvc will not delete the volume in heketi, and pv keeps Released status, how can i delete them all if I only delete pvc?

[jarrpa]

How are you creating your PVCs? Does kubectl delete pv <PV_NAME> work?

[yangyuw]

@jarrpa this is my pvc yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-class: gluster-heketi
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
  creationTimestamp: 2017-11-25T15:54:52Z
  name: a
  namespace: cicd
  resourceVersion: "146377"
  selfLink: /api/v1/namespaces/cicd/persistentvolumeclaims/a
  uid: f990e2f2-d1f8-11e7-bde6-080027330f35
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  volumeName: pvc-f990e2f2-d1f8-11e7-bde6-080027330f35
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  phase: Bound

kubectl delete pv <PV_NAME> works but also can not delete volume in heketi, so if i want to delete a volume, i need call API to delete pv, pvc, heketi volume, that sucks.

[jarrpa]

What's the YAML definition for your StorageClass and one of your PVs?

[yangyuw]

@jarrpa
pv:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.beta.kubernetes.io/gid: "2001"
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/glusterfs
  creationTimestamp: 2017-11-25T16:03:11Z
  name: pvc-1f10da0e-d1fa-11e7-bde6-080027330f35
  resourceVersion: "147571"
  selfLink: /api/v1/persistentvolumespvc-1f10da0e-d1fa-11e7-bde6-080027330f35
  uid: 22b847e7-d1fa-11e7-bde6-080027330f35
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: b
    namespace: test
    resourceVersion: "147552"
    uid: 1f10da0e-d1fa-11e7-bde6-080027330f35
  glusterfs:
    endpoints: glusterfs-dynamic-b
    path: vol_4b6e6578f4cc9f9f090727e9e8ba211d
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gluster-heketi
status:
  phase: Bound

storageclass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: 2017-11-24T13:47:02Z
  name: gluster-heketi
  resourceVersion: "2943"
  selfLink: /apis/storage.k8s.io/v1/storageclassesgluster-heketi
  uid: f385d847-d11d-11e7-bde6-080027330f35
parameters:
  resturl: http://192.168.0.43:8085
  restuser: joe
  restuserkey: My Secret Life
provisioner: kubernetes.io/glusterfs

[jarrpa]

Hmm... try to delete a bound PVC, then check kubectl logs <HEKETI_POD> to see if there's any logs of the delete activity.

[yangyuw]

@jarrpa
most of time delete pvc, pv, heketi volume will be deleted at the same time(heketi volume late about one minute), I don't know how to make this bug, just find there are few garbage volumes.

[jarrpa]

So you're saying that most of the time when you delete a PVC the PV and heketi volume get successfully deleted as well? Check the heketi logs anyway and look for any errors. Offhand, I'm guessing you might be able to reproduce this if you delete more than one PVC in rapid succession.

[yangyuw]

maybe I do need call API 3 times to delete one volume. (sad face)

[yangyuw]

@jarrpa ok , I'll try, thanks for your help

[xiaomage008]

@jarrpa I also encountered the same issue. After pvc,pv delete successfully, but volume still existed on heketi. I have to delete them on heketi one by one? Any good idea for this issue. thanks a lot.

[xiaomage008]

@yu-yang2 I also face the same issue. After pvc & pv deleted, but volume still existed, i have to delete them one by one using heketi-cli volume delete order. Have you found the root reason?

[yangyuw]

@xiaomage008 I haven't meet this issue after upgrade heketi to 5.0.

[jarrpa]

Yes, older versions of heketi have issues where the volumes between k8s and heketi go out of sync. This should be remedied by upgrading to newer versions.

[maurya-m]

@jarrpa , having a similar issue where in using heketi-cli v8.0.0-334-g39f7df22, wherein when i delete my pvc (gets deleted) pv remains in Failed status ( after released status a while) , current using gluster 4.1.7 version.

Checking the logs (heketi) the volume does not exit , so the delete went fine! , but the communication back to the pv status is lost i guess, (also checked with any pods are attached) . geeting this message in the pv logs :

Message: server did not provide a message (status 500: Internal Server Error)

Setup :
heketi version : heketi-cli v8.0.0-334-g39f7df22
glusterfs 4.1.7
K8s : v1.13.4

Any ideas why this happening?

thanks,
Maurya

[zhanghuidinah]

/reopen

[amgads]

We are facing the same problem when undeploying an application with tens of pvcs/pvs. It looks like pvc and pv are gone.
kubectl get pvc or kubectl get pv ---> no resources found

But both volumes in heketi as well as - the corresponding glusterfs-dynamic services are not still showing up -- not removed:
heketi-cli volume list or heketi-cli topology info --> show the volumes there. Also gluster volume list
and for dynamic services:

kubectl get svc --all-namespaces

NAMESPACE NAME TYPE CLUSTE
R-IP EXTERNAL-IP PORT(S) AGE
default glusterfs-dynamic-067b2708-8ffc-4f39-a5ee-67b05675fb73 ClusterIP 10.254
.1.187 1/TCP 3d18h
default glusterfs-dynamic-08ae158a-0064-42ed-b24b-8713186379ef ClusterIP 10.254
.160.55 1/TCP 3d18h
default glusterfs-dynamic-125a572f-d9a1-4ce1-b948-f1c1676cec16 ClusterIP 10.254
.180.65 1/TCP 3d18h
.....

[amgads]

/reopen