RAM isn't freed after a request in my Cloud Run service?

[adrianom]

Apologies for all the Cloud Run questions, but I stumbled across something that is puzzling me. I have a Cloud Run service that I ran about 3 requests with. The memory allocation doesn't seem to be freed after each request. Each peak seen in the graph is a request and it only goes up as I add a new request. I assumed that after a request, RAM would drop down and then spike up as a request is processed, but it only goes up until it eventually errors out for lack of RAM. I was told by a Google staff fellow (I'm going to guess this person is reading this right now) that cleaning up after a request is recommended because Cloud Run uses an in-memory file system (more info here) so I wrote some code to delete the all the files that my request generates. Even so, this general increase in RAM is happening. Is this normal? Maybe I'm not cleaning up after my requests correctly? My Cloud Run service is using 1 CPU with 512 MB of RAM. I also just recently learned about Cloud Logging, so I'm going to dive deeper into the logs of my service to see what I can learn and get back to this post.

 

[adrianom]

I'm more convinced that I'm just not accounting for all the files my Cloud Run service is generating. This is another graph of an entirely different Cloud Run service. And as I expected Cloud Run memory to work, it peaks when a request is processed and once it's done, it lowers back to a baseline memory usage until the next request (or the next peak you see in the graph) occurs. Disregard the pink line below the peaks. More to probably come...

[kolban · Accepted solution]

It certainly feels like a "memory leak".  If we imagine a request being received, the request is processed by a Docker image that you have written.  If the memory usage continues to grow, that says to me that the memory consumption of a container instance keeps growing.  Now we need to ask ourselves "how is this happening?" ... two immediate thoughts:

1. You are explicitly allocating memory in your app and not releasing it before the end.  You might be doing this explicitly (memory allocations) or you might be doing it implicitly ... for example ... if you have a global variable that persists between calls and you keep "adding more data into it" (for example an array) then the memory consumption would grow.

2. Cloud Run has an unexpected property ... and that is that files that are written to are stored in memory rather than disk.  This means that any writes you perform consume memory.

The burning question becomes ... how to diagnose such problems?  If I were sitting in your seat, what I would do is fire up a Cloud Run instance by making a request to it.  If memory serves, this means that it remains even after the request has finished for some period of time (15 minutes?).   I'd then run command on that instance to determine how much memory is being used.  I'd then send in some more requests and see if the memory consumption grows as a function of requests.   Now, I likely said something off there ... I said "run command on Cloud Run instance".   There is a recipe that we can use to do that .. but .. be careful.  Set up credentials so that only authorized users can run command ... basically don't give public access to your Cloud Run during testing ... otherwise ANYONE could login.   A recipe for logging in can be found here:
https://binx.io/2020/04/25/how-to-login-to-a-google-cloud-run-container/ - This recipe I haven't personally tried.

The one I used is found here ... https://medium.com/@kolban1/shell-exec-with-cloud-run-fbc6d299f6d4

Its work to set up these tests but will likely grow your skills and knowledge.  Another notion, would be to run the container locally under Docker.  Send in requests to your local instance.  Since it is running under Docker, you can create a login shell and your are good to go.

[adrianom]

Broken record at this point, but thanks so much again!