Tcpdump packet capture for a specific pod in RKE2

Tested on:

• RKE2 cluster v1.31
• Calico CNI
• Cilium CNI

A similar approach may be possible for other CNIs

There is a requirement, example if asked by support, to capture network traffic for a specific pod using tcpdump.

1. Identify the pod whose traffic needs to be captured and take note of the node it is running on.

$ kubectl get pods -o wide -A
NAMESPACE             NAME                                                   READY   STATUS      RESTARTS       AGE    IP              NODE                      NOMINATED NODE   READINESS GATES
...
kube-system           rke2-coredns-rke2-coredns-797bf7dd6-w7wsc              1/1     Running     0              134m   10.42.86.195    cobalt60-wn-6g7rj-54q8q   <none>           <none>
...

In this case, pod rke2-coredns-rke2-coredns-797bf7dd6-w7wsc is running on node cobalt60-wn-6g7rj-54q8q .

2. ssh to the node running the pod and check that tcpdump is installed.

cobalt60-wn-6g7rj-54q8q:~ # which tcpdump
/usr/sbin/tcpdump

3. Identify the container(s) associated with the desired pod.  Make sure the selected hex ID is that for the container, not the image or pod.

cobalt60-wn-6g7rj-54q8q:~ # CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml /var/lib/rancher/rke2/bin/crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                            ATTEMPT             POD ID              POD
...
8aa3d0032fc9a       48e9296167c66       2 hours ago         Running             coredns                         0                   f9bdc6b0a59bd       rke2-coredns-rke2-coredns-797bf7dd6-w7wsc
...

4. Get the system pid for the process running in identified container.

cobalt60-wn-6g7rj-54q8q:~ # CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml /var/lib/rancher/rke2/bin/crictl inspect --output go-template --template '{{.info.pid}}' 8aa3d0032fc9a
4162

5. Determine the network namespace for that process.  Note that pods, such as etcd, that use "hostNetwork: true" do not use a separate network namespace.

cobalt60-wn-6g7rj-54q8q:~ # ip netns identify 4162
cni-7419a4ad-67af-0e7e-2df2-6fe7f7b77e4c

6. Identify the network interface associated with that namespace.

cobalt60-wn-6g7rj-54q8q:~ # ip link show | grep -B 1 cni-7419a4ad-67af-0e7e-2df2-6fe7f7b77e4c
9: cali7e96ee7b80c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-7419a4ad-67af-0e7e-2df2-6fe7f7b77e4c

7. Run tcpdump on the identified interface.  The command line options will depend on your requirements.

cobalt60-wn-6g7rj-54q8q:~ # tcpdump -n -i cali7e96ee7b80c
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cali7e96ee7b80c, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:12:18.964206 IP 10.1.0.14.53536 > 10.42.86.195.8181: Flags [S], seq 777492885, win 64860, options [mss 1410,sackOK,TS val 1113708317 ecr 0,nop,wscale 7], length 0
13:12:18.964206 IP 10.1.0.14.43598 > 10.42.86.195.8080: Flags [S], seq 855374987, win 64860, options [mss 1410,sackOK,TS val 1113708317 ecr 0,nop,wscale 7], length 0
13:12:18.964222 IP 10.42.86.195.8181 > 10.1.0.14.53536: Flags [S.], seq 3609130794, ack 777492886, win 64308, options [mss 1410,sackOK,TS val 3762933600 ecr 1113708317,nop,wscale 7], length 0
13:12:18.964222 IP 10.42.86.195.8080 > 10.1.0.14.43598: Flags [S.], seq 1247134848, ack 855374988, win 64308, options [mss 1410,sackOK,TS val 3762933600 ecr 1113708317,nop,wscale 7], length 0
13:12:18.964240 IP 10.1.0.14.53536 > 10.42.86.195.8181: Flags [.], ack 1, win 507, options [nop,nop,TS val 1113708317 ecr 3762933600], length 0
13:12:18.964240 IP 10.1.0.14.43598 > 10.42.86.195.8080: Flags [.], ack 1, win 507, options [nop,nop,TS val 1113708317 ecr 3762933600], length 0
13:12:18.964325 IP 10.1.0.14.53536 > 10.42.86.195.8181: Flags [P.], seq 1:110, ack 1, win 507, options [nop,nop,TS val 1113708317 ecr 3762933600], length 109
13:12:18.964325 IP 10.1.0.14.43598 > 10.42.86.195.8080: Flags [P.], seq 1:111, ack 1, win 507, options [nop,nop,TS val 1113708317 ecr 3762933600], length 110: HTTP: GET /health HTTP/1.1

The above example was taken on RKE2 with a Calico CNI.  Below shows example interface naming from a similar cluster running Cilium instead:

strontium90-wn-rk2bv-76whs:~ # ip link show | grep -B 1 cni-b5a1c8dd-8231-270d-a144-bd707514a036
18: lxc0ce172ff24fc@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 9e:9b:dd:16:64:37 brd ff:ff:ff:ff:ff:ff link-netns cni-b5a1c8dd-8231-270d-a144-bd707514a036

strontium90-wn-rk2bv-76whs:~ # tcpdump -n -i lxc0ce172ff24fc
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lxc0ce172ff24fc, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:39:44.021209 IP 10.42.1.62.45172 > 10.42.1.35.8080: Flags [S], seq 1208423489, win 64240, options [mss 1460,sackOK,TS val 3285512146 ecr 0,nop,wscale 7], length 0
15:39:44.021209 IP 10.42.1.62.54102 > 10.42.1.35.8181: Flags [S], seq 1541630679, win 64240, options [mss 1460,sackOK,TS val 3285512146 ecr 0,nop,wscale 7], length 0
15:39:44.021228 IP 10.42.1.35.8080 > 10.42.1.62.45172: Flags [S.], seq 1625494060, ack 1208423490, win 65160, options [mss 1460,sackOK,TS val 2403526766 ecr 3285512146,nop,wscale 7], length 0
15:39:44.021228 IP 10.42.1.35.8181 > 10.42.1.62.54102: Flags [S.], seq 4130180126, ack 1541630680, win 65160, options [mss 1460,sackOK,TS val 2403526766 ecr 3285512146,nop,wscale 7], length 0