Availability during node maintenance

← Documentation · Welcome

Nodes get rebooted: Talos and Kubernetes upgrades, hypervisor maintenance, the occasional crash. The platform drains a node before a planned reboot, so your pods are evicted and rescheduled - but an evicted pod is gone until its replacement is ready. Whether your users notice is decided by a few lines in your manifests. The example app isg-example ships all of them; copy from there.

Two replicas

autoscaling:
  minReplicas: 2       # or spec.replicas: 2 without an HPA

With one replica your app is down for one pod start on every node maintenance. Two replicas fit the default quota if your requests are modest (the example requests 50m CPU / 64Mi per pod).

Spread the replicas over different nodes

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: my-app     # your pod labels

Without it the scheduler may put both replicas on the same node. The cluster injects a soft constraint of this kind into pods that declare neither topologySpreadConstraints nor podAntiAffinity; declaring your own is still better, because you control the selector.

Stop gracefully

lifecycle:
  preStop:
    sleep:
      seconds: 5       # native hook, no shell or sleep binary needed

Kubernetes removes a stopping pod from the Service and sends SIGTERM at the same moment; the ingress controller needs a second to notice. The pause keeps the process serving until no new requests arrive. Your process must exit on SIGTERM within terminationGracePeriodSeconds (default 30 s). The cluster injects this hook into containers that have no preStop of their own.

PodDisruptionBudget: the safe shape

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: my-app

maxUnavailable: 1 lets the platform evict one pod at a time and can never block a node drain. Do not use minAvailable: minAvailable: 1 on a single-replica app blocks every node drain until someone deletes the PDB; the cluster's policy report flags such PDBs (audit-pdb-blocks-drain).

Apps that cannot run twice

Single-writer databases, SQLite-backed apps, Socket.IO without a shared adapter: run one replica, no PDB (or maxUnavailable: 1), and accept one pod restart per node maintenance. For PostgreSQL use a CloudNativePG Cluster with instances: 2 or more; the operator switches over instead of going down.