Shared in-memory volume not working anymore

[varazi-attercop]

Hi there!

tl;dr: Mounted volume shared between the ingress and the worker container in Cloud Run suddenly stopped working.

We have a setup where the ingress container uses an in-memory volume mounted at /tmp to store incoming uploaded files, which are afterwards picked up by a worker container that mounts the same volume at its own /tmp path.

This used to work until a few days ago, Friday IIRC; now it's broken and we've been trying all kinds of things to troubleshoot the issue with no success so far.

Has anyone ran into the same or similar problem? Any hints?

P.S. I'm aware it's a preview feature, but I can't see any notice about it being degraded/disabled/discontinued.

[knet]

This is definitely not expected, we're working on bringing this feature to
GA and it's expected to be stable.

I assume that you've checked that both containers mount the same volume?
What kind of error are you getting?
-------------------------------------------
Karolina Netolicka
Product Manager, Serverless
knet@google.com

[varazi-attercop]

Here's what I have from the console:

The error that I'm getting is (it's a python app):

DEBUG:root:Saved file to /tmp/out_1722272886887.pdf

[2024-07-29 17:08:07,189: ERROR/ForkPoolWorker-8] [Errno 2] No such file or directory: '/tmp/out_1722272886887.pdf'

"kombu.exceptions.EncodeError: Cannot serialize FileNotFoundError(2, 'No such file or directory')"

The messages above are not sequential; I picked them to show that the file is indeed created and then it's not found by the other container.

We tested this with a GCS type volume to verify that the file is indeed created and to check the file attributes; we're getting the same error with that too.