How to troubleshoot using the namespace of a container

Task

When troubleshooting an issue, often a faithful reproduction and exact environment are needed. This can be a challenge in a containerized environment, where tools and a shell environment may not be easily available within containers of a Pod.

Steps

There are two approaches that can be taken:

Sidecar container

By running a container in the same namespaces as another, it's possible to use that container for troubleshooting.

The sidecar container can be started using the same network and PID namespaces while attaching the same volumes:

• Set the ID or name of the container you wish to troubleshoot:

ID=<container ID or name>

• Run the sidecar container using the network, PID and volumes

docker run -it --net=container:$ID --pid=container:$ID --volumes-from=$ID alpine sh

• It is now possible to troubleshoot with commands from the alpine container, within the context of the container or Pod with the issue.

For example, if you were experiencing a network issue from this Pod, it is now possible to use tools available in the sidecar container to simulate the connection, view the network configuration and troubleshoot interactively.

Substitute the alpine container as needed with an image of your choice.

Note, this will attach the same volumes as the parent container, but the parent container read/write layers will not be accesible - to access the same container filesystem, see the nsenter example below.

Use the host tools with nsenter

Alternatively you can use tools available on the host for the same usecase with the nsenter command. The nsenter command is standard on most Linux distributions, for example on Ubuntu it is provided by the util-linux package.

• Set the ID or name of the container you wish to troubleshoot:

ID=<container ID or name>

• Obtain the first process in the container (PID 1):

PID=$(docker inspect --format '{{ .State.Pid }}' $ID)

• Run commands available on the node within the context of all of the container/Pod namespaces with nsenter:

nsenter -a -t $PID <command>

For example, if troubleshooting a network issue, tools from the node like tcpdump, curl, dig and mtr can be used to troubleshoot the issue interactively.

Note, the -a flag is available in recent versions of nsenter, if this does not succeed, use a flag for a specific namespace, check the nsenter --help output.