Cloud Function - Stuck forever and failed Deployment. How to release?

[vbgcp]

I created a Cloud Function but for some reason, it is stuck and seems like in continious loop and never completed build:

Its been more than a day and it is stuck in loop. How to kill it and release the resources?

Its unfortunate that Google lets this happen, whatever is the issue, it should exit out of it.

How do kill this and remove the resources?

[kolban]

Howdy vbgcp,

I'm sorry this has happened to you.  Let's see if we can't figure out the cause and get it fixed.   Hopefully we will find more diagnostics than what we see in the Console.  When we deploy a Cloud Function, we are telling Google "Hey .. here is a chunk of my code ... when an incoming request arrives, please send that request to my code and you take care of running it".  The act of giving your code to Cloud Functions is called a "deployment".  Now, if the code that is supplied to Cloud Functions contains an error, is missing a dependency or some other technical reason that Cloud Functions can't initiate an instance, it will (hopefully) tell us.  And the message it may give us is "Failed to deploy".  At this point, we need to figure out what has gone wrong ... debugging.   When a deployment occurs, Google Cloud logs all the steps of the deployment in Cloud Logging.  That would be where I would go first.

I would suggest we open up Cloud Logging's log explorer in the project and lets see what happened the last time we tried a deployment.  If all goes well, we'll see a clear error message and get an "aha" moment, fix the error, redeploy and we'll be back on track.

[vbgcp]

Hi @kolban 

Well, only thing I can see is of indentation error due to which, it could not compile:

It is not able to pull the code for me to make update:

content:

Upon looking at storage location, I see the archive:

So, not sure where the problem if I am unable to fix the error. Can we kill the process or any other alternative?

@vbgcp 

[kolban]

Howdy vbgcp.  I think we are getting closer.  I'm sensing there is a problem in the source of the Python code as indicated in the logs.  Yeah!!!  That should be easily fixable.  In our thread, we haven't described how we are deploying the Cloud Function.  What recipe did you use to create the Cloud Function in the first place?  Let's see if we can't capture that and take it apart and see what we can do next.

[vbgcp]

I have done via Inline code and did not use any tool on my desktop. Is this time shown my local time (IST) or Region time (EST) I do not see they are matching:

Logs : 

[kolban]

We can always delete a Cloud Function deployment (failed or succeeded).  I'm tempted to suggest we do the following.

• Start following the steps and created a brand new Cloud Function definition.
• For each step, screen shot it and paste it in a Google Cloud Doc so that you can refer to it later and share it with support (if needed)
• When all the steps have been completed ... pause and step back and ask "Are we seeing everything we expected to see and is it all working?".  If yes, end of story ... if no, then now we have a 100% paper trail of everything we did to get here.

From there we can examine each step in turn.   My hope is that once we look at the log steps you follow, we will see "Aha ... this is where I supply my source code" ... and then, when you need to update the source (eg. to fix a typo or bug or indent problem) ... we will immediately be able to say "Its there on step 4 we need to change".

[vbgcp]

I have already created a new function and it is working fine.  I know the problem with the code and we fixed it and its working. No issue there.

My main point is, how to delete or fix this function which is in loop or stuck somewhere. Its surprising that GCP is behaving this way for a Indent issue (in other cases, if there was a Indent issue, it failed deployment and I was able to fix the code). But here, deployment failed and stuck in loop and does not allow me to delete too.

This part is the frustrating part. So, the main question is, how can I delete the function (I do not care about the code) and release resources (if any)

[kolban]

Oh OHHH ... I am so sorry.  I missed the "looping" story ... your posts were clear it was my interpretation/reading that failed us.  I am sensing that we are saying that the console is saying that the deployment is just "stuck".  If I were with you, the first thing I'd do is refresh the browser just in case it is a bug in the UI alone and it is had gotten mismatched state.  Some of our UIs show "busy" until an outcome is received from Google Cloud.  If the outcome event "got lost" it may still show busy.

Next, if that doesn't change things, I'd open up a Cloud Shell and we do some listings and actions.

For example:

gcloud functions list

to get a list of our deployed functions.  Does the "broken" deployment show up there?

If yes, then 

gcloud functions delete <NAME> --region <region>

and delete it.  Let's try those and see where we end up.

[vbgcp]

Well, that did not help. Looks like it never got deployed and technically does not exist?

[kolban]

But it does show up in a listing of the Cloud Functions?   Let's maybe look at its details:

gcloud functions describe <NAME> --region <REGION>

[vbgcp]

@kolban Here is the output:

[kolban]

It looks like this puzzle has been seen before:

• https://issuetracker.google.com/244960362

I also searched some Google support cases and found one dated 2023-03-06 with identity: case 43822050

It presented exactly as yours does.  The core of the case was:

----

At the moment, the inconvenience you are experiencing with the Cloud Functions 2nd Gen being hung in a broken state seems to be related to a known issue with Cloud Run deployments we were reported of.

After further reviewing your project’s Cloud Functions, I’ve noticed both have this error message when selecting them and clicking on “Show More” for the first alerting message: “Cloud Run service for the function was not found. The function will not work correctly. Please redeploy”.

The Engineering team has worked on a new patch to avoid new deployments from facing this same issue, and has informed that the best solution is redeploying the services. Because of this, could you click on any of these functions and then proceed to the “Edit and Redeploy” button showing on the “This function has failed to deploy and will not work correctly. Please edit and redeploy” message?

----

To follow up with the issue you are facing, if the button is still grayed out, could you try to redeploy via the Google Cloud CLI? You may find helpful information in this link [1]. To deploy the function with an HTTP trigger, run the following command in the directory that contains the sample code:

gcloud functions deploy [YOUR_FUNCTION_NAME] \
--gen2 \
--runtime=nodejs16 \
--region=us-east1 \
--source=. \
--entry-point=[YOUR_CODE_ENTRYPOINT] \
--trigger-http \
--allow-unauthenticated

*If you require authentication, please omit the [--allow-unauthenticated] flag.

This will help us “unlock” the hung Cloud Functions from your project, and after this you should be able to deploy and monitor your Functions as you desire, not only from the CLI.

----

Following this was a response from the customer that it worked.

[vbgcp]

@kolban 

Finally, I think I was able to delete the cloud function. 

The option "Edit and Redeploy" was disabled. So, next option was to go with redeploy via command line. But when I did try to redeploy , I had the below problem with the source code:

So, I went to the storage container and looked for the code version history and "restored" from the oldest version:

The moment I did it, in few seconds, the function build (stuck situation) got into failed situation:

When going to details of the function, I see that "Edit and Redeploy" is now enabled:

Now, I am able to go to the Code and I can now make the necessary code changes as needed or I can even delete the function if I want to:

So, ultimately, looks like if you are in this situation, you can go to the version history of the function and restore to a prior version before the function went into a loop and restore to that version and go back to your function and make the necessary fixes or delete the function.

Thanks a lot @kolban for your patience in assisting me to go through the thought process. It helped me learn few more things.

[kolban · Accepted solution]

And THANK YOU for your patience and the extremely clear responses you posted.