Dataplane v2 - Cilium and iptables

[slawx]

Found that for example CILIUM_FORWARD and CILIUM_OUTPUT_raw chains in iptables have rules for interfaces lxc+, while such don't exist in the dataplane v2., gke+ can be found:

$ ip -br link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0 UP 42:01:0a:80:00:2b <BROADCAST,MULTICAST,UP,LOWER_UP>
docker0 DOWN 02:42:7a:18:c1:62 <NO-CARRIER,BROADCAST,MULTICAST,UP>
cilium_net@cilium_host UP 2a:59:ba:e6:35:1d <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP>
cilium_host@cilium_net UP 26:d9:2f:af:3b:45 <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP>
gke28dda7e5534@if2 UP ba:bc:27:f3:0e:be <BROADCAST,MULTICAST,UP,LOWER_UP>
gkec3b8ab422a1@if2 UP de:1f:2a:52:97:28 <BROADCAST,MULTICAST,UP,LOWER_UP>
gkebacabba5ca7@if2 UP 0e:4d:fe:9e:88:11 <BROADCAST,MULTICAST,UP,LOWER_UP>
gke4caf43461d6@if2 UP b6:7e:f9:94:48:78 <BROADCAST,MULTICAST,UP,LOWER_UP>
gke021bec221f9@if2 UP 1a:72:b4:8c:d1:9e <BROADCAST,MULTICAST,UP,LOWER_UP>
gke21461b5a532@if2 UP 7e:9d:e8:74:f6:ac <BROADCAST,MULTICAST,UP,LOWER_UP>

Chain CILIUM_FORWARD (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * cilium_host 0.0.0.0/0 0.0.0.0/0 /* cilium: any->cluster on cilium_host forward accept */
0 0 ACCEPT all -- cilium_host * 0.0.0.0/0 0.0.0.0/0 /* cilium: cluster->any on cilium_host forward accept (nodeport) */
0 0 ACCEPT all -- lxc+ * 0.0.0.0/0 0.0.0.0/0 /* cilium: cluster->any on lxc+ forward accept */
0 0 ACCEPT all -- cilium_net * 0.0.0.0/0 0.0.0.0/0 /* cilium: cluster->any on cilium_net forward accept (nodeport) */
0 0 ACCEPT all -- * lxc+ 0.0.0.0/0 0.0.0.0/0 /* cilium: any->cluster on lxc+ forward accept */
0 0 ACCEPT all -- lxc+ * 0.0.0.0/0 0.0.0.0/0 /* cilium: cluster->any on lxc+ forward accept (nodeport) */

Chain CILIUM_OUTPUT_raw (1 references)
pkts bytes target prot opt in out source destination
0 0 CT all -- * lxc+ 0.0.0.0/0 0.0.0.0/0 mark match 0xa00/0xfffffeff /* cilium: NOTRACK for proxy return traffic */ CT notrack
0 0 CT all -- * cilium_host 0.0.0.0/0 0.0.0.0/0 mark match 0xa00/0xfffffeff /* cilium: NOTRACK for proxy return traffic */ CT notrack
0 0 CT all -- * lxc+ 0.0.0.0/0 0.0.0.0/0 mark match 0x800/0xe00 /* cilium: NOTRACK for L7 proxy upstream traffic */ CT notrack
0 0 CT all -- * cilium_host 0.0.0.0/0 0.0.0.0/0 mark match 0x800/0xe00 /* cilium: NOTRACK for L7 proxy upstream traffic */ CT notrack

Can anyone confirm that's ok. Why do rules with non-existing interfaces exist in iptables? Was it intended? If yes, then when is that used?

Thank you

[VannGuce]

Possibly, in Kubernetes, the interface might be created dynamically because pods are usually scheduled and removed. In the iptables rules possibly created temporarily when the pods are started and removed if they are stopped. I believe this is not common since interfaces usually appear or disappear when workload scales up and down.

Let me also share with you this link[1] about the use of cillum in iptables.

[1]https://medium.com/coccoc-engineering-blog/a-note-on-ciliums-iptables-usage-39b4709fa584

[slawx]

I brought this up as with installed cilium on envs with kubernetes have by default interface names and iptables rules matching those interfaces "lxc". Here in GKE interfaces are called gke and cilium rules are still with lxc names, those rules are never hit and interfaces are never lxc. This is something I've been struggling with for the last few days as wasn't able to push traffic for protocols other than sctp/tcp/udp, changing these interface names solved it for me.

[slawx]

Hello,

During new cluster setup with multiple interfaces and networks found that lxc interfaces get created for additional pod interfaces. That would mean iptables rules are correct.

I found FORWARD rules allowing only tcp/udp/icmp/sctp,

$ sudo iptables -vnL FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
...
83 10992 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
32 4940 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT sctp -- * * 0.0.0.0/0 0.0.0.0/0

how can I add more protocols to be allowed? Can you shed more light how can I allow forward traffic between gke interfaces?
Would like to add for example:
0 0 ACCEPT 112 -- * * 0.0.0.0/0 0.0.0.0/0

Thank you