Pod with nodeName hangs while waiting for PVC to be provisioned

[enrichman]

Creating a pod without the nodeName being specified hangs the provisioning of the PVC, because of the WaitForFirstConsumer volume binding mode.

Test

1. Simple Pod and PVC works 👍

apiVersion: v1
kind: Pod
metadata:
  name: sample
spec:
  containers:
  - image: "debian:stable-slim"
    name: sample
    command: ["/bin/bash", "-c", "--"]
    args: [ "while true; do sleep 30; done;" ]
    volumeMounts:
      - name: vol
        mountPath: /tmp/test
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: myclaim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: "10M"

2. Specifying the nodeName where the Pod should be scheduled hangs 🔴

PVC log:

waiting for first consumer to be created before binding

apiVersion: v1
kind: Pod
metadata:
  name: sample
spec:
  nodeName: k3d-test-server-0
  containers:
  - image: "debian:stable-slim"
    name: sample
    command: ["/bin/bash", "-c", "--"]
    args: [ "while true; do sleep 30; done;" ]
    volumeMounts:
      - name: vol
        mountPath: /tmp/test
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: myclaim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: "10M"

Fixes

Two possible fixes:

1. set the same node in the PVC with the volume.kubernetes.io/selected-node annotation
2. change the storageclass volumeBindingMode to Immediate

References

I think this is related, so not sure if this is an actual problem/bug, or not.

[brandond]

I believe this is documented upstream as a limitation:
https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode

If you choose to use WaitForFirstConsumer, do not use nodeName in the Pod spec to specify node affinity. If nodeName is used in this case, the scheduler will be bypassed and PVC will remain in pending state.
Instead, you can use node selector for kubernetes.io/hostname

[enrichman]

Thanks!