Cloud Function Stops Execution

[mightymouse3062]

I have a situation where there is a Typescript/NodeJS based Cloud Function that runs a large number of Firestore database reads (about 15 million) in a short period of time (about 5 minutes). It also generates a bunch of logs (couple hundred thousand). The function is configured with a 5 minute timeout and if run normally, takes less than a minute to complete. There are cases where several hundred invocations are triggered around the same time (metrics instance count scales to about 300). 

When I have the large peak of invocations I am finding that there are some executions that just stop. In the logs I can filter on one of the executions that just stop and the logs just end. There is no final message of `Function execution took XXX ms, finished with status: 'ok'`. 

I have read through https://cloud.google.com/functions/docs/troubleshooting#unexpected-stops and have not found anything in the code that the solutions could cover. 

Based on the first and last log timestamps, the function is running for about 180 seconds, which is below the 300 second timeout setting. I do not see any error messages in the logs (again, the logs just stop). 

Are there any suggestions on how I can get the missing logs to figure out what is going on with my Cloud Function 

Thank you. 

PS: I know my function/architecture is in need of a lot of optimization, it's on my list to fix 🙂 

[ianmckee]

Hey @mightymouse3062 ,

I'm assuming you're invoking your cloud function via an event-driven architecture, in which, there's a strict 9 minute timeout period. This could be why your function is stopping after a 180 second (3 minutes runtime) runtime, followed by a 5 minute timeout (8 minutes total runtime). By this point, attempting another 3 minute runtime would exceed the 9 minute timeout period. You could try invoking your cloud function using a HTTP functions as this has a 60 minute timeout period, or like you said, you could focus on optimization if you feel it could be improved!

Hope this helps!

[mightymouse3062]

Thank you for the followup on this topic. 

I am invoking it via an event driven architecture.

The interesting part about this issue is not receiving a final message from Cloud Functions, the logs just completely stop. I wasn't able to determine if it hit the max timeout of 9 minutes or not. 

The current hack I found to avoid this issue is limit the maximum number of concurrent executions for this Function to 10. Not ideal but everything runs.