net_cls in the time of cgroup v2

Do you ever wonder how to apply specific networking rules to a chosen program, e.g. a browser?
Network rules based on PID (process ID) were dropped in 2005 for good reasons. Let’s not bring up entire containers for a moment (where you can define separate configuration in network_namespaces(7)) but take only one part of them, namely control groups (cgroups).

Rules with net_cls

Back in the day, you could use net_cls cgroup controller which allows applying rules to a subset (group) of processes. You create a cgroup, associate a class id with it and utilize that in your iptables or nftables rules.

mkdir /mnt/net_cls/special
echo 0x10001 >/mnt/net_cls/special/net_cls.classid

(Note /mnt/net_cls is a mount of a separate cgroup hierarchy, unlike the unified v2 hierarchy.)
The example below shows how to log packets to or from processes in the given cgroup (via class id):

iptables -A OUTPUT -m cgroup --cgroup 0x10001 -j MARK --set-mark 10
iptables -A OUTPUT -m mark --mark 10 -j LOG --log-prefix "CGROUP.net_cls "

[ 993.068082] [ T4468] CGROUP.net_clsIN= OUT=ens3 SRC=10.100.53.79 DST=10.100.2.10 LEN=54 TOS=0x00 PREC=0x00 TTL=64 ID=24502 DF PROTO=UDP SPT=35693 DPT=53 LEN=34 MARK=0xa

Or in nftable parlance:

nft add rule ip filter output \
meta cgroup 0x10001 \
meta mark set 10 \
log prefix \"CGROUP.net_cls\"

[ 1627.690357] [ T5272] CGROUP.net_clsIN= OUT=ens3 SRC=10.100.53.79 DST=10.100.2.10 LEN=54 TOS=0x00 PREC=0x00 TTL=64 ID=50601 DF PROTO=UDP SPT=36838 DPT=53 LEN=34 MARK=0xa

The apparent advantage of the net_cls hierarchy is that you specify the network configuration once and mere migration of processes into/from the cgroup affects the networking behavior.

echo $PID >/mnt/net_cls/special/cgroup.procs

Enter the modern world where an app is rarely a single process.
Furthermore, the default v2 hierarchy does not provide the net_cls controller. So how to achieve similar results now? The crucial thing to realize is that dynamism in cgroup v2 is not based on process migrations but on rather static cgroup tree whose attributes are modified in order to exercise or adjust the control.

Rules with unified hierarchy

The conversion of the two examples above would thus start with finding where our process resides (or launch it in a known cgroup) and then apply the rules for those cgroups.

iptables -A OUTPUT -m cgroup --path user.slice/user-0.slice/session-5.scope -j MARK --set-mark 20
iptables -A OUTPUT -m mark --mark 20 -j LOG --log-prefix "CGROUP.v2path "

nft add rule ip filter output \
socket cgroupv2 level 3 "user.slice/user-0.slice/session-5.scope" \
meta mark set 20 \
log prefix \"CGROUP.v2path\"

(The socket may belong to a cgroup arbitrarily deep in the hierarchy. The level 3 predicate allows extracting only the subpath of the socket’s cgroup to match (counting from root down).)

The addition (or removal) of rules to (or from) cgroups replace process migrations to achieve the dynamism. There is the burden on the administrator keep track of the applied rules, which net_cls eliminated by implicitly storing this information in the process group membership. However, the migration of running processes to apply configuration is inherently racy and herding a multi-process application is clumsy. The latter is solved elegantly on modern Linux desktops where each application is placed into its cgroup in advance and the networking rules applied to that cgroup ensure every process is properly ruled. (For comparison, Android executes all apps under different users so network filtering may be based on UID (user ID) predicates.)

It’s worth reiterating that migrations are not modus operandi in cgroup v2 and migrating a process between two cgroups will not affect rules applied to already created sockets (those remain associated to their original cgroup).

Assessment

The method with net_cls allowed delegation of the switching operation to non-privileged users — they would not be able to change network rules but they might be authorized to carry out process migrations between selected cgroups. The direct application of networking rules to a v2 cgroup is a privileged operation so it cannot be delegated so trivially. True delegation would be possible in a net namespace owned by the user (user_namespaces(7)). The cgroup v2 is successor to v1, so the migration-less approach is more future-proof.

Further alternatives

Another option how to achieve the dynamism would be to equip each cgroup with a custom eBPF program (hook into BPF_PROG_TYPE_CGROUP_SKB). (Attaching such programs is a privileged operation by default.)

See also

Motivated by Mullvad VPN split tunneling feature and the removal of v1 code in openSUSE Tumbleweed.