mounting a CS bucket on Workbench instance

[marcousescloud]

Hi everybody,

I've succesfully mounted a CS bucket on Workbench from both JupyterLab Interface and JupyterLab terminal (with "gcsfuse").

The mounted bucket works well: I can write, read, delete and so on my python files.

There's a problem anyway: when I stop and restart the instance, I can see the name of the mounted bucket, but I can't access my files from both the JupyterLab interface and the terminal.

It seems like at restart time the bucket is unmounted by the system. If this supposition was true, the mounting facility would be completely unuseful.

Can someone help me to understand what's going on and suggest some solution?

Thank you all.

Marco

[dawnberdan]

Hi @marcousescloud,

Welcome to Google Cloud Community!

You're experiencing an issue where a GCS bucket mounted with gcsfuse is not automatically reconnected after restarting your Workbench instance. To ensure consistent access to your GCS bucket, you'll need to set up an automated process for mounting it at startup.

gcsfuse mounts your GCS bucket as a local filesystem, which is temporary and doesn't persist across reboots. To maintain access, you need to automate the mounting process.

1. Create a Startup Script:

• Create a shell script (e.g., mount-gcs-bucket.sh) with the following content:

  #!/bin/bash
  # Script to mount GCS bucket using gcsfuse
  
  # Create a directory for the mount point if it doesn’t exist
  mkdir -p /path/to/mount/point
  
  # Mount the GCS bucket
  gcsfuse my-bucket /path/to/mount/point​

• Replace my-bucket with your bucket name and /path/to/mount/point with the local directory where you want to mount the bucket.
• Make the script executable: you can refer to this documentation for additional insights on scripting.

  chmod +x /path/to/mount-gcs-bucket.sh

2. Add the Script to Startup (Workbench-Specific):

• Workbench Startup Hooks: Refer to the Google Cloud Workbench Documentation for configuring startup hooks. Typically, you can set this up via the GCP Console or within Workbench settings.
• Workbench Startup Scripts: If Workbench supports custom startup scripts, place mount-gcs-bucket.sh in the designated location.

3. Add the Script to Startup (Linux Instance):

• Using /etc/rc.local: Add this line to /etc/rc.local before the exit 0 line.

  /path/to/mount-gcs-bucket.sh​

• Using systemd: Create a systemd service file (e.g., /etc/systemd/system/gcs-mount.service)

  [Unit]
  Description=Mount GCS Bucket
  After=network-online.target
  Wants=network-online.target
  
  [Service]
  Type=simple
  ExecStart=/path/to/mount-gcs-bucket.sh
  Restart=on-failure
  
  [Install]
  WantedBy=multi-user.target

• Reload systemd, enable, and start the service

  sudo systemctl daemon-reload
  sudo systemctl enable gcs-mount
  sudo systemctl start gcs-mount

• You can refer to this documentation for more details.

4. Verify Permissions and Configuration:

• IAM Roles: Ensure your Workbench instance has the appropriate IAM roles (e.g., Storage Object Viewer, Storage Object Admin) to access the GCS bucket.
• Credentials: Verify that the necessary credentials (e.g., service account keys) are available for gcsfuse to authenticate.

5. Check Logs and Errors:

• For systemd Services: Use sudo journalctl -u gcs-mount.service to view service logs.
• For /etc/rc.local: Check general system logs or any specific logs added to your script.

Additional Considerations:

• Alternative Mount Points: Consider using mount points like/mnt or /media for your GCS bucket.
• Alternative Tools: Explore tools like rclone or cloud storage libraries for different use cases.
• Security: Ensure that credentials used for mounting are securely managed. You can refer to Google Cloud Security Best Practices for more information.

I hope the above information is helpful.

[marcousescloud]

hi dawnberdan,
thank you for your kind and detailed answer and sorry for the delay in answering.

Following your suggestions, I've tried a number of different ways and configurations.

At the end of the day, I've found that the following work well for me:

gcsfuse --implicit-dirs --dir-mode "777" --file-mode "777" -o allow_other "CS_bucket" "path"

Thank you again.