If you read the official Kubernetes documentation, init containers look straightforward. They run to completion, they run sequentially before your application containers start, and if they fail, the pod restarts. It looks like a five-line YAML addition and you are good to go.

But once you ship an init container to production, you quickly discover that failure has a very specific meaning, sequence order comes with a catch, and resource accounting works in a way that the docs gloss over.

The first time an init container got OOMKilled in a pod I was on call for, I spent forty minutes debugging the main application container. It was only when a teammate pointed out the initContainerStatuses block that I realized where the error actually was. This post is the guide I wish I had back then.

The 90-Second Refresh

Before we get into the tricky details, let us review how init containers are supposed to work.

Init containers run sequentially. Each one must run to completion before the next one starts, and all of them must finish before any of your main application containers can begin. They share volumes with the pod but run in separate process spaces. If any init container fails, the kubelet restarts the entire pod according to your restartPolicy settings, which means the whole sequence starts over from the beginning.

This mental model is fine for basic setups, but it leaves out five critical production behaviors.

A quick note on Kubernetes 1.29 and newer: Native sidecar containers, which are init containers with restartPolicy: Always, became beta in 1.29 and stable in 1.33. They solve the classic deadlock where an init container blocks waiting for a service mesh sidecar. This post focuses on traditional init container behavior, which is still the default in most clusters. If you are using a newer cluster version, the upstream sidecar docs are worth a look.

1. Resources Do Not Add Up the Way You Expect

Kubernetes does not schedule pods by adding up the resource requests of every container in the spec. Instead, the scheduler calculates the pod request using a specific rule. The kubelet looks at two numbers and takes the higher value:

  • The sum of all app container requests.
  • The highest request of any single init container.

To see why this matters, consider a pod with three app containers that request 256 MiB of memory each, and one init container that requests 2 GiB.

Kubernetes Init Container Resource Scheduling Diagram

The scheduler looks for a node with 2 GiB of free memory, not 2.75 GiB. This makes physical sense because the init container finishes before the app containers start, so they do not need the memory at the same time.

Keep in mind that native sidecar containers are an exception to this rule. Because they remain running alongside your app containers, their resource requests are added directly to the sum of your main container requests.

But this behavior creates a subtle trap. If the node runs low on memory while the init container is running, the container gets OOMKilled. Because this terminates the entire pod, the failure shows up as a pod-level status. If you only look at the main app container logs, you will not see any errors.

To verify this, look directly at the status block for the init containers:

kubectl get pod <pod> -o jsonpath='{.status.initContainerStatuses[*].lastState}{"\n"}'

If you see OOMKilled in the output, you hit this scheduling gotcha. Your options are to increase the node size, reduce the init container request by refactoring what it does, or move the initialization work out of the pod entirely.

2. A Failed Init Container Restarts the Whole Pod

When an init container fails, Kubernetes does not just restart that single container. It restarts the entire pod, which resets the initialization sequence.

This is fine if all your init containers are idempotent, meaning they can run multiple times without causing side effects. It becomes a problem when they are not.

Imagine this scenario: your first init container fetches credentials from Vault and writes them to a shared emptyDir volume. Your second init container runs a database migration. If the database migration is flaky and exits with a non-zero code, the pod restarts.

When the pod restarts, the first init container runs again. It connects to Vault, requests a new token, and overwrites the shared volume. This can flood your Vault audit logs, trigger rate limits, or fail completely if you use one-time tokens.

You cannot assume your migrations or setup scripts will never fail. Instead, you need to design your init containers so that repeating the entire sequence is safe and inexpensive.

3. The chown -R Init Container Will Eventually Slow You Down

A common pattern for setting file permissions on shared volumes looks like this:

initContainers:
- name: fix-perms
  image: busybox
  command: ['sh', '-c', 'chown -R 1000:1000 /data']
  volumeMounts:
  - name: data
    mountPath: /data

This works well at first. But once your persistent volume grows to hundreds of thousands of files, chown -R has to walk the entire directory tree on every startup. This can block the pod from starting for ten or twenty minutes. During this time, your rollouts stall and your deployment status commands hang.

A better way is to use the fsGroup setting in the pod security context. The kubelet applies this group ID when mounting the volume. You can optimize this further by setting fsGroupChangePolicy to OnRootMismatch, which graduated to stable in version 1.23. This policy only runs the recursive permission change if the root directory permissions do not match the target group:

spec:
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch

Many Helm charts do not include this policy by default. Adding it manually saves you from stuck deployments when scaling stateful workloads with large volumes.

4. Init Containers Can Silently Deadlock

A classic deadlock occurs when an init container needs to call an external service through a service mesh, like Istio or Linkerd. The service mesh sidecar is an application container, which means it will not start until all init containers have completed. The init container waits forever for the network path to open, and the sidecar waits for the init container to finish.

The pod remains stuck in the Init:0/1 state without raising any explicit errors.

In clusters older than version 1.29, you can work around this by bypassing the mesh for the init container (using host network settings or egress rules) or by moving the check into the main application startup script. In newer versions, native sidecar containers resolve this problem.

A similar deadlock happens with dependency checks:

- name: wait-for-postgres
  image: busybox
  command: ['sh', '-c', 'until nc -z postgres 5432; do sleep 1; done']

Without a timeout, a database outage or a DNS misconfiguration will block your deployments indefinitely. You should always cap these loops:

command: ['sh', '-c', 'timeout 60 sh -c "until nc -z postgres 5432; do sleep 1; done"']

If the database is not ready within 60 seconds, the container fails. A pod showing CrashLoopBackOff is much easier to detect and debug than a pod stuck in Init:0/1 forever.

5. Successful Init Container Logs Are Easily Lost

You can inspect init container logs using the container flag:

kubectl logs <pod> -c <init-name>

However, this only works while the pod is running. If the pod restarts due to a node drain, an application crash, or a new deployment, the Kubernetes garbage collector will eventually clean up the old containers, taking their logs with them.

While you can use the --previous flag to look at the logs of the last run:

kubectl logs <pod> -c <init-name> --previous

This only keeps one generation of history. If your init container did something important, like running a database migration or populating a cache, and the pod restarts twice, you lose that history.

To prevent this, avoid relying solely on standard output logs for critical initialization steps. You should ship these logs to a central collector, write them to a durable path, or include clear log identifiers that you can query in your log aggregator later.

Production-Ready Patterns

If you use init containers, these three patterns are usually worth the extra complexity:

  • Wait-for-dependency with a timeout: This is helpful if your application crashes unpredictably when dependencies are missing. Blocking in the init phase makes the dependency issue visible in the pod status.
  • Secret materialization: You can run an init container that fetches secrets from a manager (like HashiCorp Vault or AWS Secrets Manager) and writes them to an emptyDir volume. The application container then reads them as local files. This keeps credentials out of environment variables and makes rotation simpler.
  • Instance-specific setup: This includes generating a unique TLS certificate from a local certificate authority, retrieving pod-specific config, or seeding a local cache before startup.

Anti-Patterns to Avoid

Keep your deployment specs clean by avoiding these common mistakes:

  • Running database migrations in a Deployment init container: If your deployment has multiple replicas, they will all try to run the migration simultaneously. They will compete for database locks, block startup, and potentially deadlock. Migrations are better suited for a one-off Kubernetes Job or a pre-deployment step.
  • Heavy CPU or disk work: Init container runtimes add directly to your pod startup latency and rolling update times. A long-running init container slows down your scaling response times.
  • Unbounded network calls: Always set timeouts on external requests to prevent startup deadlocks.
  • Using recursive chown on large volumes: Use the native fsGroup configurations instead.

A Debugging Checklist

When a pod is stuck in the Init state or restarting frequently, you can debug it using these commands:

# 1. Identify which init container is stuck and view its state
kubectl get pod <pod> -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{": "}{.state}{"\n"}{end}'

# 2. Check the logs of the current init container run
kubectl logs <pod> -c <init-name>

# 3. View logs from the previous run if the container restarted
kubectl logs <pod> -c <init-name> --previous

# 4. Check for exit codes, OOMKills, or general errors
kubectl describe pod <pod> | grep -A 20 "Init Containers"

# 5. Review scheduling events and resource constraints
kubectl describe pod <pod> | grep -A 10 "Events"

The first command is especially helpful, it shows you the exact state of every init container on a single line, saving you from scrolling through pages of kubectl describe output.

Wrap Up

Init containers look simple on paper, but their resource calculations, restart rules, and interaction with sidecars require careful planning in production. By keeping your init containers fast, idempotent, and bounded by timeouts, you can make your deployments resilient and keep your on-call shifts quiet.