Drain Kubernetes Pods Before a Rolling Update Removes Them

• Readiness controls new Service traffic; it does not finish work already inside the Pod.
• Handle the stop signal in the application, stop accepting work, and finish active requests within the grace period.
• Treat preStop as part of that same time budget, then verify the real rollout and error rate.

A Kubernetes Deployment can keep enough replicas available and still cut a request. Replica counts govern capacity. They do not tell your application how to finish work that reached an old Pod.

Graceful shutdown needs one ordered contract between the Service path, the process and the rollout. Build that contract first.

Give readiness one clear job

A readiness probe answers whether a container should receive new Service traffic. When it fails, the EndpointSlice controller removes the Pod IP from matching Service endpoints.

Do not reuse a broad liveness check without thought. Liveness decides whether Kubernetes should restart the container. Readiness can go false during startup, overload or shutdown without asking for a restart.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 1

This is a labelled pattern, not a setting proven for your service. The endpoint should fail when the process has begun draining and should not return ready until dependencies and warm-up work can support real traffic.

The service-worker rollout guide handles old browser clients. Here the unit being replaced is a server Pod, and the handoff happens through Service endpoints.

Handle the stop signal in process 1

During normal Pod termination, the kubelet asks the container runtime to stop the main process. Containerd and CRI-O use SIGTERM by default unless the image defines another stop signal.

Your application should catch that signal. Mark readiness false, stop taking new jobs, close listeners in the right order, and wait for active work to finish. Put a deadline on the wait so the process can exit cleanly.

If a shell wrapper is process 1 and does not forward signals, the application may never see SIGTERM. Use an exec-form container command or a small init that forwards signals and reaps children.

Do not claim that every request will survive. Node failure, force deletion and hard process crashes can skip the polite path. Queue consumers still need idempotent work; the webhook retry guide shows the same principle at an HTTP boundary.

Budget preStop inside the grace period

Kubernetes runs a configured preStop hook before it sends the normal stop signal. The hook does not receive a separate timer. Its runtime counts inside terminationGracePeriodSeconds, which defaults to 30 seconds.

spec:
  terminationGracePeriodSeconds: 45
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/app/begin-drain"]

The 45-second value and command are examples. Set the budget from measured request, job and connection times. If the hook consumes most of it, the application gets little time after the stop signal before a forced kill.

A short hook can tell the application to drain or allow endpoint changes to spread. A blind sleep can hide a race, slow every rollout and still be too short for one load balancer. Prefer a state change you can observe.

Read the terminating endpoint state

Terminating endpoints can stay in EndpointSlices for a while. Kubernetes marks their ready condition false for compatibility. A client that supports connection draining can inspect the separate serving and terminating conditions.

That detail matters when an ingress, service mesh or external load balancer has its own health checks and propagation delay. Verify what your actual data path reads. Do not assume a Pod leaving readiness instantly removes it from every upstream.

The IPv6 launch checklist uses the same discipline: prove the entire traffic path, not only one control-plane record.

Set rollout capacity after shutdown works

A Deployment's rolling strategy uses maxUnavailable and maxSurge to limit unavailable and extra Pods. The defaults are 25% for each. Kubernetes does not count terminating Pods as available, and they can temporarily push resource use above the expected surge total.

For a small replica set, calculate the rounded values instead of trusting percentages. Make sure the cluster has room for the new Pod and for an old Pod that is still draining.

Use the container digest guide so a rollback names exact bytes, then watch kubectl rollout status, availability and application errors together.

Run one controlled drain test

1. Generate steady test traffic. Include at least one request or job near the normal long end.
2. Start a rollout. Record when the old Pod becomes terminating and when its endpoint conditions change.
3. Watch the process. Confirm it receives the stop signal and stops accepting new work.
4. Check outcomes. Look for resets, 5xx responses, duplicate jobs and work that ran past the deadline.
5. Test rollback and pressure. Repeat with full replica load and enough cluster capacity for surge plus draining Pods.

These are procedures for your own cluster. TechWeb did not deploy the manifest, inspect your ingress or measure your application.

Frequently asked questions

Is a readiness probe enough for graceful shutdown?

No. It can stop new Service traffic, but the application must still finish active work and respond correctly to its stop signal.

Does preStop add time beyond the grace period?

No. The hook normally runs inside the Pod's termination grace period. Give the hook and application enough shared time.

Should every Pod use a sleep in preStop?

No. A delay can help a known propagation gap, but measure the real path and prefer an observable drain action over a blind wait.

Sources

Kubernetes: Pod Lifecycle · Kubernetes: Pods and Endpoint Termination Flow · Kubernetes: Deployments · Kubernetes: Probes. Check the documentation for your cluster release and data path before changing production.