GKE Image Streaming Not Functioning as Expected - Size limit?

[nkem]

Summary: In a GKE Standard cluster with Image Streaming enabled, my (largish) container image stored in Artifact Registry within the same region is not utilizing Image Streaming as anticipated.

Following the steps described in Schedule a workload using Image streaming, I can see image streaming works correctly for the sample image at us-docker.pkg.dev (327 MB). My own image at us-central1-docker.pkg.dev (900 MiB) keeps getting pulled without streaming (no "ImageStreaming" message). Is this an undocumented size restriction that was recently added? Because I am fairly sure this used to work at the end of 2023 with 4 GiB large images. Am I missing anything else?

Environment Details:

• GKE Cluster Tier: Standard
• GKE Cluster Mode: Standard
• GKE Cluster Version: 1.30.6-gke.1125000
• Node Image Type: cos_containerd (cos-113-18244-236-26)
• GKE Cluster Region: us-central1
• Artifact Registry Region: us-central1 / us-central1-docker.pkg.dev
• Private Nodes: Disabled
• VPC Service Controls: None
• Custom Service Account: Yes

Requirements Checklist:

• [X] Container File System API: You must enable the Container File System API.
• [X] GKE Version: GKE Standard clusters must run version 1.18.6-gke.4801 or later.
• [X] Node Image: You must use the Container-Optimized OS with containerd node image.
• [X] Artifact Registry Location: Verify that the Artifact Registry repository is in the same region as the GKE nodes or in a corresponding multi-region.
• [ ] Private Google Access: If private nodes are enabled, ensure Private Google Access is configured on the subnet.
• [ ] VPC Service Controls: If applicable, include the Image Streaming API (containerfilesystem.googleapis.com) in the service perimeter.
• [X] Service Account Permissions: For custom service accounts, confirm that the roles/serviceusage.serviceUsageConsumer IAM role is granted.

Limitations to Consider:

• Container images using the V2 Image Manifest, schema version 1, are ineligible for Image Streaming.
  Using schema version 2: "application/vnd.docker.distribution.manifest.v2+json"
• Images encrypted with customer-managed encryption keys (CMEK) require GKE version 1.25.3-gke.1000 or later for Image Streaming support.
  Does not apply / GKE version 1.30.6-gke.1125000
• Images with duplicate layers are unsupported and will not utilize Image Streaming.
  Neither duplicate nor empty layers present
• Workloads that access numerous files during initialization may experience increased startup times due to remote file read latency.
  Event log should still show an "ImageStreaming" message for this particular image
• The benefits of Image Streaming may not be evident during the initial image pull; caching improves performance in subsequent pulls.
  Event log should still show an "ImageStreaming" message for this particular image on subsequent pulls
• GKE Standard clusters use the cluster-level configuration to determine whether to enable Image  streaming
  Image streaming is enabled on Cluster level and for every newly generated node pool
• Linux file capabilities such as CAP_NET_RAW are supported with Image Streaming in GKE version 1.22.6-gke.300 and later.
  Does not apply / GKE version 1.30.6-gke.1125000

Troubleshooting Steps:

1. Verified Image Eligibility:

   • Container image does not use the V2 Image Manifest, schema version 1
   • No duplicate or empty layers within the image

2. Checked Cluster and Node Configuration:

   • Confirmed that Image Streaming is enabled at both the cluster and node pool levels.
   • Verified that nodes are using the Container-Optimized OS with containerd.

3. Validate Service Account Permissions:

   • Service Account has Service Usage Consumer IAM role (also tried with default service account and full scope)

4. Monitored Pod Initialization:

   • kubectl events shows several system helper pods with ImageStreaming, but not my image

5. Reviewed Logs for Errors:

   • Nothing shows up related to gcfsd, no errors

[nkem · Accepted solution]

Fixed it. Short answer: The docker image contained a file named `\`. I noticed it because image streaming worked fine with the base image, so I went step by step through the layers and eventually step by step through the remaining copied directories/files until image streaming stopped working. Fun.