NetworkPolicy for GCSFuse CSI bucket mounting

[John_Liakh]

Greetings, everyone!

I'm trying to figure out network (port, IP address, dns name)  requirements GCSFuse CSI driver has.

I am guided by the instruction and everything is working as expected as long as I don't use network policy. But with network policy, weird things happen: pod get's stuck on CreateContainer stage, hangs in this state until some timeout and throws:

kubelet Error: context deadline exceeded
kubelet Error: failed to reserve container name "<NAME>": name "<NAME>" is reserved for "<HASH>"

Also, gke-gcsfuse-sidecar logs are stuck at:

Opening GCS connection...
Creating a mount at "/dev/fd/3"
Creating a temporary directory at "/gcsfuse-t
Creating a new server...
Set up root directory for bucket <BUCKET_NAME>

It is missing the successful mounting confirmation, which happens without NetworkPolicy:

...
Mounting file system "<BUCKET_NAME>"...
File system has been successfully mounted.
...

Logs of the gcs-fuse-csi-driver container of the same node:

I1016 utils.go:83] /csi.v1.Node/NodePublishVolume called with request: volume_id:"<BUCKET_NAME>" target_path:"/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/<VOLUME_NAME>-pv/mount" volume_capability:<mount:<mount_flags:"implicit-dirs" mount_flags:"uid=1000" mount_flags:"gid=1000" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:"csi.storage.k8s.io/ephemeral" value:"false" > volume_context:<key:"csi.storage.k8s.io/pod.name" value:"<POD_NAME>" > volume_context:<key:"csi.storage.k8s.io/pod.namespace" value:"<NS>" > volume_context:<key:"csi.storage.k8s.io/pod.uid" value:"<POD_ID>" > volume_context:<key:"csi.storage.k8s.io/serviceAccount.name" value:"<SA>" > volume_context:<key:"csi.storage.k8s.io/serviceAccount.tokens" value:"***stripped***" >
I10161 node.go:248] NodePublishVolume succeeded on volume "<BUCKET_NAME>" to target path "/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/<VOLUME_NAME>-pv/mount", mount already exists.
I1016 utils.go:89] /csi.v1.Node/NodePublishVolume succeeded.
I1016 utils.go:83] /csi.v1.Node/NodeUnpublishVolume called with request: volume_id:"<BUCKET_NAME>" target_path:"/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/<VOLUME_NAME>-pv/mount"
I1016 mount_linux.go:791] Unmounting /var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/<VOLUME_NAME>-pv/mount
W1016 mount_helper_common.go:142] Warning: "/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/<VOLUME_NAME>-pv/mount" is not a mountpoint, deleting
I1016 node.go:306] NodeUnpublishVolume succeeded on target path "/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/<VOLUME_NAME>-pv/mount"
I1016 utils.go:89] /csi.v1.Node/NodeUnpublishVolume succeeded.
I1016 utils.go:83] /csi.v1.Node/NodeGetCapabilities called with request:
I1016 utils.go:89] /csi.v1.Node/NodeGetCapabilities succeeded.

Also, I've checked the 

"/var/lib/kubelet/pods/<POD_ID>/volumes/kubernetes.io~csi/<VOLUME_NAME>-pv/mount path on the node and it was not completely mounted. Any command to this path, like ls, touch, mkdir hangs up. It seems like it has failed to mount a bucket on the node.

I've checked node logs for the kubelet, contained, and fuse csi pods and see no errors related to this event.

In the policy, i've allowed access to the k8s api, and gcsfusecsi pod.

The only idea I have ATM is to try to trace a connection with tcpdump...

I will be very thankful for any advice.

[VannGuce]

Hi,

Based on “pod gets stuck on CreateContainer stage” one of its causes is network related issue wherein I believe comes from the deployment of your NetworkPolicy since it was working when NetworkPolicy is disabled.

For this concern I suggest to use this link[1] as a guidance when deploying NetworkPolicy to check if we missed out anything. For diagnosing about stuck pods we will get more information if we run the these debugging tools “kubectl describe pod”, “kubectl logs” and “kubectl exec”

Enable also the logging for GCSFuse CSI driver,GCSFuse sidecar and kubelet to get more information about pod’s creation and GCS bucket mounting, the information that you will gather from this diagnosis will help you to configure your NetworkPolicy properly.

[1]https://reuvenharrison.medium.com/an-introduction-to-kubernetes-network-policies-for-security-people...

[garisingh]

There are a couple of options here:

1) You can try to use FQDN Network Policy (which is currently in preview) and allow egress to "storage.googleapis.com"

2) You'd need to use Private Google Access in order to use standard network policy and allow egress to a specific CIDR range (the public IPs for Google services change).

- Make sure PSA is enabled for the VPC  subnet(s) used by your GKE cluster

- Set up a private DNS zone for the PGA VIP

- Allow egress to the CIDR range "199.36.153.8/30"

[sigi31415]

I actually was hoping to use this setup to solve a problem we face: I want to make sure that a POD is isolated but has access to one specific GCS Bucket. To do so we use a web proxy and whitelist the path the bucket needs.

Was there a particular reason why this is implemented as a fuse driver sidecar instead of a storage provider?

We probably will allow it through a NetworkPolicy and deny it through a firewall in the container but will probably go with FQDN Network Policy later.

The huge security risk i can see is that GCS scales very well. The chance that an attacker creates an anymous GCS bucket and can push a lot of data out of a system is in my opinion a valid attack vector.

[John_Liakh · Accepted solution]

Sorry for delay. For some reason, googclecloudcommunity became unavailable with my ISP.

With TCPdump running on the node with workload on it I was able to detect the connections to the

metadata.google.internal http

It resolved to the 169.254.169.254
Checks with netcat showed that metadata.google.internal was inaccessible.
So i've added this IP with 80 port to the network policy. And nothing changed - it was unreachable still.

Some googling the docs drove me to the: network-policy-and-workload-identity 
Adding 169.254.169.254/32 80 to the network policy solved the issue.

Thanks for Your replies and recommendations!