Debugging with Ephemeral Containers

Debugging applications in Kubernetes can be tricky. Containers are designed to be small, immutable, and purpose-built. That is great for production, but not always ideal when something breaks. Many production images are minimal or distroless. They may not include tools that are useful for troubleshooting.

In some cases, the application container may already be crashing, which means kubectl exec is not useful. In other cases, accessing the nodes may not be possible. Since Pods are immutable, it is impossible to add another container for troubleshooting.

This is where ephemeral containers help.

What Are Ephemeral Containers?

In Kubernetes, ephemeral containers are a special type of container designed to run temporarily inside an existing Pod. Their primary purpose is to help administrators and developers troubleshoot, inspect, and debug live applications without disrupting the running service. They are not meant to run application workloads. Instead, they are designed for operational debugging.

An ephemeral container is not added to a Pod by editing spec.containers. Kubernetes treats it differently from regular containers. When an ephemeral container is created, a request is sent to the Kubernetes API server using the Pod’s ephemeralcontainers subresource. This distinction is important because most of a Pod’s spec is immutable after creation; Kubernetes does not allow you to simply edit a running Pod and append a normal container to spec.containers.

Key Characteristics of Ephemeral Containers

Unlike standard containers, ephemeral containers have the following characteristics:

1. They do not have guaranteed CPU or memory resources (limits or requests).
2. If an ephemeral container crashes or completes its task, it will never restart.
3. They do not include fields such as ports, livenessProbe, or readinessProbe.

Examples

For testing, the Percona Operator for MySQL based on XtraDB Cluster is installed. The installation steps can be found here. Once the installation is complete, the running pods should look similar to the following:

NOTE: It is important to note that this behavior depends on your environment, specifically whether you have the required privileges or Security Context Constraints (SCC) in place. The commands below are for demonstration purposes and do not necessarily follow all best practices, such as avoiding generic ubuntu images, long-running shells like bash, or running containers as root.Always run ephemeral containers with a strong focus on security, especially in production systems.

Let’s look at some examples of how ephemeral containers can be useful.

1. Ephemeral Container with shared network namespace

When an ephemeral container is created in a Pod, it shares the network namespace of all other containers in that Pod. This is particularly useful for troubleshooting network-related issues.

Let’s create an ephemeral container named debug-1 in the MySQL pod cluster1-pxc-0:

When we check the processes in the container, only the bash process that was started when the container was created is running.

However, port 3306 is open because the MySQL process in the primary container shares the same network namespace as our debug container.

This capability is highly effective for analyzing network traffic, egress connectivity, and local service availability.Let’s examine the Pod spec and status to see how ephemeral containers are represented.

The following is the spec of the ephemeral container. Note the securityContext, which we will discuss later in this post:

Status of ephemeral containers

2. Ephemeral Container with shared network namespace, pid namespace

While sharing the network namespace is useful, sharing the PID namespace to inspect running processes can be beneficial in many debugging scenarios.

Let’s create an ephemeral container named debug-2 in the cluster1-pxc-0 Pod, specifically targeting the PID namespace of the pxc container:

A key change from the previous command is the addition of the --target=pxc flag. This creates an ephemeral container that shares the PID namespace of the pxc container.

When we check the processes in the container, we can see the MySQL processes.

We can also verify this by network stats

3. Ephemeral Container with shared network namespace, shared pid namespace, shared volume

In some cases, you may need to collect dumps, check database files, or inspect logs, which requires access to the filesystem. However, each container has its own mount namespace, and mount namespaces cannot be shared directly.A volume mounted in a Pod, however, can be shared across containers, including ephemeral containers.

Let’s check the volumes present in the cluster1-pxc-0 pod and how they are mounted to the pxc container.

As seen above, the persistent volume holding the database files is mounted at /var/lib/mysql.

Let’s create an ephemeral container named debug-3 in the cluster1-pxc-0 pod, sharing the PID namespace of the pxc container and mounting the datadir volume at /db-mount:

The command above creates the ephemeral container in the pod. To access it, we will attach to the running debug container:

As demonstrated, the database files located at /var/lib/mysql are now accessible at /db-mount within the debug-3 container.

Since the volume was mounted with the “readOnly”: true parameter, no writes can be performed.

If you require write permissions, simply omit the “readOnly”: true flag.

Now, attach to the container and create a file in the volume mount:

4. Ephemeral Container on a Kubernetes node’s namespace and filesystem

If you need to examine system logs (such as kernel logs or dmesg) but do not have SSH access to the nodes, you can run an ephemeral container directly on the node’s namespace and filesystem.

Let’s check the nodes of the Kubernetes cluster and run an ephemeral container directly on one:

Run an ephemeral container:

Node’s filesystem can be accessed at /host.

Behind the scenes, a Pod with the name node-debugger-<node-name>-<hash> is created.

Let’s check the spec of the debug pod

Some key observations from the spec are the following: 

The specifications above indicate excessive permissions for a regular application; consequently, these might be disabled by your system administrator via security policies.

Profile of an Ephemeral Container

An interesting option for the kubectl debug command is --profile.

The official documentation provides the following:

These profiles define the capabilities associated with the SecurityContext and determine how the host’s filesystem and namespaces are accessed. The security context details for each profile are listed below; further details are available in the source code.

The pod’s execution behavior depends on the profile chosen when running the kubectl debug command.

Caveats with Ephemeral Containers

Even though ephemeral containers are useful, there are several caveats and potential security risks that users should be aware of:

1. Ephemeral containers may be able to inspect processes, network traffic, environment variables, mounted volumes, or service account context depending on the Pod and cluster configuration.
2. They can bypass the security benefits of distroless or minimal images.
3. Ephemeral containers can expose secrets
4. Once an ephemeral container is added, it remains part of the Pod spec; it is not possible to remove it.
5. As long as the ephemeral container’s main process is running, you can attach to it using kubectl attach.
6. Ephemeral containers behaviour is unpredictable, containers might terminate abruptly depending on the resource configuration and utilization of the pod.

Guardrails and Best practices with Ephemeral Containers

Ephemeral containers are powerful, but they can create operational and security risks if used carelessly. Adhering to the following guardrails and best practices can help mitigate these risks:

1. Control ephemeral containers access through RBAC. Only necessary users should have the privilege.
2. Avoid using the sysadmin profile when possible. The restricted profile is better suited for maintaining a strong security posture.
3. Always use approved, scanned images that contain only the tools required for debugging, rather than generic images like ubuntu.
4. Avoid running debug containers with long-running processes like sleep infinity or bash. Instead, run specific tools or commands that perform the required action and then terminate.
5. Audit ephemeral containers usage.
6. Recycle Pods that contain ephemeral containers whenever possible (e.g., during maintenance windows), especially if the ephemeral process has not terminated.

Conclusion

Ephemeral containers are a powerful troubleshooting tool for Kubernetes workloads, especially when application images are minimal, distroless, or missing debugging utilities. They allow engineers to inspect a running Pod without rebuilding the image or restarting the workload.

However, they should be treated as controlled operational access, not as a default debugging shortcut. Ephemeral containers can expose sensitive runtime details such as processes, environment variables, mounted volumes, and network state.

Always restrict usage with proper RBAC. Use approved debug images, prefer the least-privileged debug profile, and reserve powerful profiles such as sysadmin for “break-glass” scenarios only.

Debug sessions should be short-lived, intentional, and tied to a real troubleshooting need. In short, ephemeral containers improve debuggability, but they must be used with clear security, audit, and operational guardrails.