Secrets Manager Access timout

[brockdavis]

How do I determine if there is an issue with Cloud Secrets Manager in the AU Region ?

My cloud run services are having issues communicating with secrets manager.

I am now getting

Total timeout of API google.cloud.secretmanager.v1.SecretManagerService exceeded 60000 milliseconds

This service has been running for months, deployed a simple update, now will not communicate with Secrets Manager.

This is in  australia-southeast1 

Any direction / help on how to determine the issue or if an outage with GCP would be helpful.

[Jb83]

I am also currently having issues with the client libraries accessing secrets from Google Secret Manager (Europe).

We are getting "502 Bad Gateway" from a majority of the requests. I thought this was related to incident https://status.cloud.google.com/incidents/VgryWZ3bxKBSUEBjXaSb#oKGDLmoneB3YgW79P7vm but it says that is resolved. Our issues started in conjuncture to this incident happening and hasn't stopped. Any ideas?

[brockdavis]

Jb83

Which version of the Client Libraries are you using?   I am using NodeJS ver 3.7.3

"@google-cloud/secret-manager": "^3.7.3", 

Hmm... going to look at upgrading.

[brockdavis]

I just created a Cloud Function that reads secrets from Secret Manager.  Running NodeJS.

I created this function in US East Coast, EU Belgium, and AU Sydney.  The function only fails within the Sydney region with Timeout.

[brockdavis]

Jb83,

My issue "can" be resolved if I reference the Project Number, not the Project ID.

Might be related to the outage you reference, not sure.

[WSevenster]

I've been using the project Number* for the last 6 months with no issues, but today the issue appeared for the first time.

[WSevenster]

I am having the same issue on the Sydney server. 

[brockdavis]

Yes, it is all related to the Sydney Cloud Run and Cloud Functions attempting to access Secrets Manager.

I have not found a solution, other than not use Secrets Manager on NEW installs.  Apparently existing deployments appear to be working, just new deployments.

[WSevenster]

I re-published a cloud function that uses it and it worked for 2 days after republishing, then got this error this morning. Hopefully it's discovered and resolved quickly, it's not mentioned on their service health page yet.
https://status.cloud.google.com/regional/asia

[dbousamra]

We are having critical issues too. All our services are Sydney based. Cloud run and Cloud Functions cannot resolve secrets.

[Neekey]

Having the same issue about 10 minutes ago, we are requesting secret manager from App Engine, it's in the Sydney region as well

[dk-outstaffer]

+1 timeouts accessing Secret Manager Service via App Engine in the Sydney region. First occurrance we can see in the logs was at 6:28am ADST. 

[sbsleb]

We are having outages on all our production accounts, timeout of 60s can't access secret manager cloud run/cloud functions, can't seem to reproduce locally.

[dk-outstaffer]

☝️ This

We can run up our app locally with no issues accessing secret manager. It only happens when deployed to App Engine.

[dbousamra]

We have seen a few successes now. Maybe 1% of requests making it through

[brockdavis]

To those affected by this, there are two solutions until Google fixes the issue.

1.  Remove use of Secrets Manager (yeah, not good solution)

2.  Move your Cloud Functions / Cloud Runs / App Engines to another region.

[dbousamra]

This is so unacceptable. No communication at all from Google.

[dk-outstaffer]

Gotta love status pages...always the last to know 😆

[dbousamra]

Are we able to be compensated for this? How does it work? I've got customers screaming at me, and my only solution is to wait. I can't easily switch regions, or hard code secrets. 

[brockdavis]

You can sign up for support:  https://console.cloud.google.com/support

After paying, you can get your incident escalated.

[sbsleb]

This is insane. No communication from them, status page says its fine, how is this acceptable for a platform provider?

[dk-outstaffer · Accepted solution]

I raised a case and received this reply a few minutes ago:

[WSevenster]

Great to have someone in here who pays for support! 😂

[karl-outstaffer]

It has been working from our side since 10:36 AEST. How is it for everyone?

[dbousamra]

Working for us since AEST 10:35

[SravanAlluri]

working for us too now

[brockdavis]

Appears to be working.  thanks.