Cloud Run Job GCSFuse Out of Memory

[mmike]

We're experiencing out of memory errors when running a cloud run job that *only* writes files to a mounted bucket. The job reads files from within the bucket, does some processing, and writes the output to the bucket.

I was under the impression that writing to the bucket would not consume the job's allocated memory, but it doesn't seem to be the case since I can see the memory consumption increasing throughout the job's lifetime (see image below). Here's the error we're getting:

time="01/08/2024 05:38:44.092331" severity=ERROR message="WriteFile: no space left on device, write /tmp/gcsfuse109880762: no space left on device"

I've run this job locally, on my machine, and the memory consumption is stable. Is the above a known issue? Does writing to a bucket consume the job's memory? If so, is there any cleanup I should be performing when writing to the mounted bucket from within a job?