Re: Mounting a GCSFuse volume with image streaming...

AvishaiW · 03-03-2025 07:36 PM

I'm trying to reduce the startup time of my Google Batch tasks. My task starts out by running the gcsfuse cli to mount a GCS bucket. When I tried using image streaming (enableImageStreaming=true) I get an error:

Error while mounting gcsfuse: mountWithArgs: mountWithStorageHandle: mount: mount: running /usr/bin/fusermount: exit status 1

This only happens when using image streaming. What could be the cause? Can you suggest a solution or a workaround?

wenyhu

Hi @AvishaiW,

Thanks for trying Batch. Could you please share your project number and one example of your failed job id to help us diagnose?

Thanks,

Wenyan

AvishaiW

project 642504272574

job id mlapp-inference-v1-ebd3f32c8c9827b1-d5772a52f88811ef8cbe5ace8a6

AvishaiW

Any updates?

AvishaiW

I've reproduced the same issue when using job.task_groups.task_spec.volumes to let Google Batch set up the mount. (job id mlapp-inference-v1-8ac905ee9dd5b564-490333fcfdda11efb06e4e82ac9)

AvishaiW

Here's a simple repro:

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "container": {
              "imageUri": "us-docker.pkg.dev/xxxxx/xxxxx/avishai-gcsfuse:01",
              "commands": [
                "/bin/sh",
                  "-c",
                "mkdir /mnt/foo && gcsfuse --log-severity trace --implicit-dirs --only-dir foo <BUCKET_NAME> /mnt/foo && ls -l /mnt/foo"
              ],
                "enableImageStreaming": true,
                "options": "--privileged"
            }
          }
        ],
        "maxRetryCount": 2,
        "maxRunDuration": "3600s"
      },
      "taskCount": "1",
      "parallelism": "0"
    }
  ],
  "allocationPolicy": {
    "instances": [
      {
        "policy": {
          "machineType": "n2d-standard-4",
          "provisioningModel": 1,
        }
      }
    ],
      "serviceAccount": {"email": "xxxx@xxxx.iam.gserviceaccount.com"}
  },
  "labels": {
    "foo": "bar1"
  },
  "logsPolicy": {
    "destination": "CLOUD_LOGGING"
  }
}

Just adding `"enableImageStreaming": true` causes the failure. The Dockerfile for the gcsfuse image:

FROM --platform linux/amd64 debian
USER root

# install GCSFuse
RUN apt-get update --allow-releaseinfo-change && \
    apt-get install sudo gnupg2 curl -y && \
    export GCSFUSE_REPO=gcsfuse-bookworm && \
    echo "deb https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list && \
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
    sudo apt-get update --allow-releaseinfo-change && \
    sudo apt-get -y install gcsfuse
# Test that it works
RUN gcsfuse --version

bolianyin

Thanks for reporting this. There is some issue with gcfuse + image streaming and the image streaming team will need more time to investigate.

If possible, please disable image streaming while gcsfuse is used as a workaround for now.

bolianyin

This is due to a limitation in the current Batch image streaming design that it cannot operate in "--priviledge" mode. We'll treat it as a feature request to address in the future. Please avoid using gcfuse and image streaming together for now.

Mounting a GCSFuse volume with image streaming on Google Batch