Linux Sysadmin Notes

Persistent Storage

4 min read

Tap Listen. On Android keep Chrome in the foreground and volume up.

Persistent Storage

Concept

Kubernetes storage is a binding problem, not a mount problem. A PersistentVolume (PV) is a piece of storage that exists in the cluster. A PersistentVolumeClaim (PVC) is a request (“I need 100Gi, ReadWriteOnce, fast-ssd”). A StorageClass is the template a provisioner uses to create PVs on demand. The Pod only mounts a bound claim.

Until the PVC is Bound and the volume is attached to the node and mounted into the Pod, the workload is not ready — no matter how healthy the Deployment looks.

Why it matters

  • Unbound PVCs are a leading cause of Pods stuck in Pending.
  • Access modes and attach limits explain “works on one replica, pending on the second”.
  • Reclaim policy decides whether deleting a PVC is a resize… or a wipe.
  • The performance you get is the backend’s performance (cloud disk, NFS, Ceph, local SSD), filtered through the CSI attach/mount path.
  • Databases on emptyDir are a restore test you did not schedule.

Mental Model

Admin or CSI provisioner
        ↓
StorageClass  ──dynamic──→  PersistentVolume
                                  ↑ bind
Pod  ──volumeMount──→  PersistentVolumeClaim

Attach (controller) → node disk/device
Mount  (kubelet + CSI node plugin) → path in the container

Static provisioning: human creates a PV; a matching PVC binds it.
Dynamic provisioning: PVC + StorageClass → provisioner creates the PV.

Access modes (what the API promises, not always what the filesystem allows):

ModeTypical meaning
ReadWriteOnce (RWO)One node at a time
ReadWriteMany (RWX)Many nodes, shared FS (NFS/CephFS-like)
ReadOnlyMany (ROX)Many nodes, read-only
ReadWriteOncePodOne Pod, period

Retain keeps the PV and data when the PVC dies (you must clean up). Delete asks the provisioner to destroy the backend volume.

Key Commands

# Cluster view
kubectl get storageclass
kubectl get pv
kubectl get pvc -A
kubectl get volumeattachment
 
# Why is this claim pending?
kubectl describe pvc <pvc> -n <ns>
kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'
 
# What did the Pod actually mount?
kubectl get pod <pod> -n <ns> -o jsonpath='{range .spec.volumes[*]}{.name}{"\t"}{.persistentVolumeClaim.claimName}{"\n"}{end}'
kubectl exec -n <ns> <pod> -- df -h
kubectl exec -n <ns> <pod> -- mount | grep -E 'pvc|nfs|rbd|csi'
 
# CSI / node side when attach fails
kubectl -n kube-system get pods | grep -iE 'csi|provisioner|attacher'
kubectl logs -n kube-system <csi-controller-pod> --tail=100
 
# Backend leftovers after a Retain
kubectl get pv -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,RECLAIM:.spec.persistentVolumeReclaimPolicy,CLAIM:.spec.claimRef.name

Common Failure Modes & Symptoms

SymptomLikely causeFirst checks
PVC Pending, no PVWrong StorageClass, quota, provisioner downSC name, provisioner logs, cloud API quotas
PVC Pending, PVs existSelector / class / access mode / size mismatchCompare PVC spec to PV spec field by field
Pod Pending: “unbound PVC”Claim not Bound yetdescribe pvc Events
Pod Pending: “node(s) had volume node affinity”RWO volume already attached elsewhere; zone mismatchVolumeAttachment, node AZ vs PV zone
Multi-attach errorTwo Pods on two nodes, one RWO diskReplica count vs access mode
Mount failed, permission deniedfsGroup, SELinux, NFS root_squash, wrong UIDdescribe pod, ls -lZ in container
Data missing after rescheduleUsed emptyDir or a new PVC each timeVolume spec; StatefulSet vs Deployment
Delete PVC, data gonereclaimPolicy: DeleteSC and PV reclaim fields
Capacity / inode full inside the volumeApplication filled the PVC, not the nodedf -h in the pod
Slow I/OTiny cloud volume, bursting exhausted, NFS latencyBackend metrics, iostat on node

Investigation Tips

  • Always pair describe pvc with describe pod. The PVC tells you binding; the Pod tells you attach/mount.
  • Zone-aware disks cannot follow a Pod to another AZ. Topology is a scheduling constraint disguised as storage.
  • StatefulSets give stable PVC identities (data-<sts>-0). Deployments do not. Do not run a single-writer database as a Deployment with a shared RWO claim and replicas: 2.
  • subPath mounts fail in ways that look like empty directories when the subPath was never created. Check the volume root.
  • Expanding a PVC is two steps: the claim size, then the filesystem (resize2fs / xfs_growfs — CSI often does this, until it does not).
  • For NFS: the server, export options, and network path are still NFS. Kubernetes will not fix soft vs hard or a wedged lock.
  • Before you delete a PVC in anger, read reclaim policy out loud.

Personal Lessons Learned

  • A Postgres “cluster” with two replicas and one RWO EBS volume spent its life with one Pod running and one Pending. That is the access mode working as designed.
  • We deleted a PVC to “recreate it bigger” on a StorageClass with Delete. The cloud disk vanished. Restore from last night’s snapshot became the change window.
  • df on the node was fine. df in the container was 100%. PVC size ≠ node disk. Look where the process writes.

Graph View

Backlinks