Adding volumes for storage, configuration, and metadata

The previous chapters focused on the pod’s containers—we shaped shopbot’s command line and environment, and claimed durable disk with a PVC. But containers are only half of what a pod typically contains. The other half is storage volumes. For an LLM serving pod that means model weights, tokenizer files, LoRA adapters, and warm caches that must outlive a single container instance—or be shared between a fetcher and the inference server. That is the focus of this chapter.

This chapter covers

• Adding a volume to a pod and mounting it into its containers.
• Persisting state across container recreation with an emptyDir volume.
• Sharing files between containers within the same pod.
• Mounting files into a container from another container image.
• Accessing the host node’s filesystem from within a pod.
• Exposing ConfigMaps, Secrets, and pod metadata through volumes.

1. Introducing volumes

A pod is a small logical computer—almost

A pod is like a small logical computer that runs a single application. The application can consist of one or more containers that run the application processes. Those processes share computing resources the way processes on one machine do: CPU, RAM, and network interfaces.

A short clarification on where those resources come from. The pod does not receive a whole virtual machine to itself. It is scheduled onto a node (often a VM or a bare-metal host). Many pods share that node. Linux namespaces and cgroups give each container an isolated view of process IDs, network, and mounts, and they cap how much CPU and memory the container may use. Disk under the node stays; what is ephemeral is the container’s own writable filesystem layer.

In a typical computer the processes would also share one filesystem—but this is where containers break the analogy. Each container has its own isolated filesystem, provided by its container image.

Think of a GPU box in a lab: the machine shares power and network (node resources), but each process still has its own private scratch disk (the container filesystem). A volume is the shared model shelf in the rack—fetcher and inference server both reach it, and it stays when one process is replaced.

Recreation, not resume

When an inference container starts, its filesystem contains what was baked into the image (runtime binaries, entrypoint)—usually not multi-gigabyte weights. Anything the process then writes (a hot-loaded LoRA, a tokenizer cache) lands in the writable layer unless you mount a volume. When the container dies for any reason—OOM, failed probe, node pressure—Kubernetes does not resume it. It recreates a new instance from the same image. The writable layer is discarded. The adapter file is gone.

That is fatal for serving: you would re-download or re-apply adapters on every crash. Mount a volume. Step through with and without one.

What mounting actually is

Definition: mounting is the act of attaching the filesystem of a storage device or volume into a specific location in the operating system’s file tree. The contents of the volume then appear at that location.

  container file tree                       volume: model-weights
  /                                        ┌──────────────────────┐
  ├── opt/vllm/       ← from image         │  model.safetensors   │
  ├── app/entry.sh    ← from image         │  tokenizer.json      │
  └── models/  ◄───────────────────────────│  adapters/…          │
       (volume contents appear here)       └──────────────────────┘
                                           mountPath: /models

Two details worth fixing in your mental model before we go further:

• The volume is declared once at the pod level (under spec.volumes); each container that wants it adds a volumeMounts entry with its own mountPath. Same volume, possibly different paths in different containers.
• Because the volume belongs to the pod, its lifetime is decoupled from any single container—that is why files survive container recreation. Whether it also survives the pod being deleted depends on the volume type; chapter 3’s PersistentVolumeClaims are the “or beyond” end of that spectrum.

When the pod starts, the volumes are created first, and then the init container is started. This is the case in all pods, regardless of whether you define the volumes before or after the init containers in the pod manifest.

  file lives in…            survives container    survives pod
                             recreation?           deletion?
  ─────────────────────────────────────────────────────────────
  container writable layer   no                    no
  emptyDir volume            yes                   no
  PVC-backed volume (ch. 3)  yes                   yes

2. Why the writable layer is not enough

Imagine an inference container that writes a small marker after it finishes loading a tokenizer cache under /tmp. Without a volume that file lives only in the writable layer. Prove the failure mode: write the file, recreate the container, watch it vanish.

apiVersion: v1
kind: Pod
metadata:
  name: llm-serve-scratch
spec:
  containers:
    - name: inference
      image: alpine:3.20
      command: ["sh", "-c"]
      args:
        - |
          mkdir -p /tmp/hf-cache
          echo "tokenizer-warm" > /tmp/hf-cache/ready.flag
          sleep 3600

$ kubectl exec llm-serve-scratch -- cat /tmp/hf-cache/ready.flag
tokenizer-warm

# force the container to exit → kubelet recreates it
$ kubectl exec llm-serve-scratch -- kill 1

$ kubectl exec llm-serve-scratch -- cat /tmp/hf-cache/ready.flag
cat: can't open '/tmp/hf-cache/ready.flag': No such file or directory

  before                            after container exits
  Pod llm-serve-scratch             Pod llm-serve-scratch  (same object)
  └─ inference                      └─ inference   NEW instance
       /tmp/hf-cache/ready.flag          /tmp/   empty again
                                         (writable layer rebuilt from image)

The node’s disk is fine. Only the container’s ephemeral layer was thrown away. Tokenizer caches, LoRA drops, and compiled graphs must live on a volume—or you pay the warm-up cost on every recreate.

3. emptyDir — a pod-lifetime folder

The simplest volume type is emptyDir. kubelet creates an empty directory when the pod starts and deletes it when the pod is removed. While the pod lives, every container recreation remounts the same directory—ideal for a per-pod model cache or adapter drop zone.

Use it for scratch weights within one pod, a writable path on a read-only root, or a shared workspace between fetcher and inference. Do not use it for the long-lived base model library—that belongs on a PVC (chapter 3) so a new pod on another node can remount the same store.

Add emptyDir and mount it

Two changes: declare the volume on the pod, then mount it where the runtime writes.

apiVersion: v1
kind: Pod
metadata:
  name: llm-serve-scratch
spec:
  volumes:
    - name: model-cache
      emptyDir: {}
  containers:
    - name: inference
      image: alpine:3.20
      command: ["sh", "-c"]
      args:
        - |
          mkdir -p /models/hf-cache
          echo "tokenizer-warm" > /models/hf-cache/ready.flag
          sleep 3600
      volumeMounts:
        - name: model-cache
          mountPath: /models

Recreate the container again. /models/hf-cache/ready.flag is still there—because it sits in model-cache, not in the writable layer.

emptyDir fields

  field        meaning
  ──────────────────────────────────────────────────────────
  medium       omit  → node disk / SSD (default)
               Memory → tmpfs (RAM; fast token caches, not durable)
  sizeLimit    cap for the directory (e.g. 10Gi for a small model tree)
               especially important with medium: Memory

volumes:
  - name: model-cache
    emptyDir:
      medium: Memory
      sizeLimit: 2Gi

In-memory emptyDir suits small, hot artefacts (tokenizer files, tiny adapters). Multi-GB .safetensors belong on disk—or better, a PVC—so you do not exhaust node RAM.

Where the folder sits on the node

For a disk-backed emptyDir, kubelet creates a normal directory and mounts it into the container. The latch path is:

  NODE
  /var/lib/kubelet/pods/<pod_UID>/volumes/
  └── kubernetes.io~empty-dir/
      └── model-cache/          ← real files live here
              │
              └── mount ──►  container:/models

  pod_UID     = metadata.uid
  model-cache = volume name in the Pod spec

$ kubectl get pod llm-serve-scratch -o jsonpath='{.metadata.uid}{"\n"}'
$ kubectl get pod llm-serve-scratch -o wide   # which node

Delete the pod and that directory goes with it. emptyDir outlives containers, not pods. Your base model registry on Ceph/NFS is a different latch (kubernetes.io~csi)—see chapter 3.

Init container downloads weights first

Volumes exist before any container runs. An init container can pull the model into the emptyDir; the inference container then starts only when the files are present.

  1. kubelet creates emptyDir model-weights
  2. init: fetch-model   downloads *.safetensors + tokenizer into the volume
  3. main: inference     loads /models and serves /v1/completions

spec:
  volumes:
    - name: model-weights
      emptyDir: {}
  initContainers:
    - name: fetch-model
      image: alpine:3.20
      command: ["sh", "-c"]
      args:
        - |
          # stand-in for: huggingface-cli download … /weights
          mkdir -p /weights
          echo "fake-weights" > /weights/model.safetensors
          echo "{}" > /weights/tokenizer.json
      volumeMounts:
        - name: model-weights
          mountPath: /weights
  containers:
    - name: inference
      image: vllm/vllm-openai:latest
      args: ["--model", "/models", "--port", "8000"]
      volumeMounts:
        - name: model-weights
          mountPath: /models
          readOnly: true

4. Sharing one emptyDir between containers

Mount the same volume in two containers at different paths. The fetcher writes where its tool expects; the inference server reads where its --model flag points. Kubernetes points both paths at one folder.

  emptyDir: model-weights
  ┌─────────────────────────┐
  │  model.safetensors      │
  │  tokenizer.json         │
  └───────────┬─────────────┘
       ┌──────┴──────┐
       ▼             ▼
  model-fetcher   inference (vLLM)
  /export/        /models/
    weights/

Analogy: one USB stick of model files. The download job mounts it as /export/weights; the GPU process mounts it as /models. Different path strings; same physical store.

  BEFORE (separate trees)                 AFTER (same emptyDir)

  model-fetcher                           model-fetcher
  /export/weights/  (empty)               /export/weights/ ──┐
                                                              ├── model-weights
  inference                                                 ┌─┘
  /models/  (missing)                     inference         │
                                          /models/ ─────────┘

spec:
  volumes:
    - name: model-weights
      emptyDir: {}
  containers:
    - name: model-fetcher
      image: alpine:3.20
      command: ["sh", "-c"]
      args:
        - |
          # stand-in for a continuous sync / pull job
          echo "weights-v3" > /export/weights/model.safetensors
          echo "{}" > /export/weights/tokenizer.json
          sleep 3600
      volumeMounts:
        - name: model-weights
          mountPath: /export/weights
    - name: inference
      image: vllm/vllm-openai:latest
      args: ["--model", "/models", "--port", "8000"]
      volumeMounts:
        - name: model-weights
          mountPath: /models
          readOnly: true

The fetcher mounts read/write. Inference mounts readOnly: true so a compromised serving process cannot overwrite the weight files. A request to POST /v1/completions loads the same model.safetensors the sidecar just wrote.

5. Image volume — ship files as an OCI image

Sometimes the “seed” is not a download script but a small image that only holds files: a tokenizer pack, chat template, or fixed LoRA. An image volume mounts that image’s filesystem into the pod. No init container. No emptyDir fill step. Volumes still come up before containers start, so kubelet pulls the volume image first.

  before (this chapter §3)              after (image volume)

  emptyDir model-extras                 volume type: image
  init: fetch-model  ──writes──►        image: acme/tokenizer-pack:1.2
  inference mounts /models/extras       mounted read-only into inference
                                        at /models/extras

spec:
  volumes:
    - name: tokenizer-pack
      image:
        reference: acme/tokenizer-pack:1.2
        pullPolicy: IfNotPresent
  containers:
    - name: inference
      image: vllm/vllm-openai:latest
      args: ["--model", "/models", "--port", "8000"]
      volumeMounts:
        - name: tokenizer-pack
          mountPath: /models/extras
          readOnly: true

kubectl describe pod shows the volume image pulled before the inference container starts—same rule as before: volumes first, then containers. Inspect the mount:

$ kubectl exec llm-serve -c inference -- ls -la /models/extras
tokenizer.json
chat_template.jinja

Version and pin the pack like any other image. Good for small immutable artefacts. Large base weights still belong on a PVC or shared model store.

6. hostPath — reach the node’s filesystem

Most app pods should not care which node they land on. System and GPU pods are different: they need this machine’s devices, drivers, or local paths. That is hostPath—a latch onto a path on the worker node’s Linux tree.

  Worker node A (H100)
  /usr/local/nvidia/          ← THIS node's drivers
  /dev/nvidia0
       │
       │  hostPath mount
       ▼
  Pod llm-serve  (scheduled here)
  container inference
    /usr/local/nvidia  ← sees node A's files

Real pattern on bare-metal GPU clusters: the NVIDIA GPU Operator (and the device plugin) mounts host paths such as NVIDIA device nodes under /dev and the driver tree (e.g. /usr/local/nvidia) into privileged system pods so containers can use the physical GPU. Another common case: a DaemonSet that tails /var/log on each node.

  NODE A disk                         Pod on A
  /usr/local/nvidia/…  ──hostPath──►  /usr/local/nvidia
  /dev/nvidia0         ──hostPath──►  /dev/nvidia0

  Reschedule pod to NODE B
  → same YAML paths, B's files/devices. Not A's.

# Illustrative — GPU Operator manages the real DaemonSet manifests
spec:
  volumes:
    - name: nvidia-driver-root
      hostPath:
        path: /usr/local/nvidia
        type: Directory
    - name: device-nodes
      hostPath:
        path: /dev
        type: Directory
  containers:
    - name: gpu-feature-discovery
      image: nvcr.io/nvidia/gpu-feature-discovery:…
      volumeMounts:
        - name: nvidia-driver-root
          mountPath: /usr/local/nvidia
          readOnly: true
        - name: device-nodes
          mountPath: /dev
      securityContext:
        privileged: true

Do not put portable model libraries on hostPath unless you pin the pod to that node. Move the pod → different disk → missing or wrong files. For shared models across nodes, use a PVC / CSI volume (chapter 3).

hostPath is also one of the most dangerous volume types. Mounting /var/run/docker.sock or the host root lets a container act as root on the node. Restrict it with admission policy; reserve it for trusted system pods (GPU Operator, log agents, CSI node plugins).

  emptyDir     pod scratch / shared model cache
  image        OCI file pack (tokenizer, template)
  hostPath     this node's drivers, devices, local paths
  next         ConfigMap / Secret / Downward API as files
  PVC (ch. 3)  portable model store across nodes

Next: ConfigMap, Secret, and Downward API volumes for serving config as files.