Persistent Cloud Run 401 Unauthorized Despite run.invoker Role and Extensive Troubleshooting

[birdma]

Hi Community,

I'm facing a persistent and baffling issue accessing a private Cloud Run service and would appreciate any insights, as I seem to have exhausted the standard troubleshooting paths.

Goal: Successfully invoke a deployed private Cloud Run service (requires authentication) using my authenticated Google Cloud user account via standard methods like curl with a gcloud access token.

Problem: After deploying the Cloud Run service configured to require authentication, attempts to access the service URL fail:

1. Browser Access: When logged into Google (tested in Incognito window), accessing the service URL results in 403 Forbidden.
2. curl with Access Token: Running the following commands in gcloud (authenticated):Consistently results in a 401 Unauthorized response page with the text "Error: Forbidden  Your client does not have permission to the requested URL / from this server."

Troubleshooting Steps Performed:

1. Deployment: Confirmed services deploy successfully via gcloud run deploy ... without the --allow-unauthenticated flag (answered 'N' to the prompt). Container logs show successful startup.
2. IAM run.invoker Role: Confirmed via gcloud run services get-iam-policy my-cloud-run-service --region=us-east1 that user definitively has the roles/run.invoker role granted on the service.
3. Unauthenticated Access: Confirmed that attempting --allow-unauthenticated fails with a FAILED_PRECONDITION error, indicating an Organization Policy correctly prevents public access via allUsers. This is expected.
4. IAP: Confirmed via Cloud Console (Service -> Security tab) that Identity-Aware Proxy (IAP) is Disabled for the service.
5. IAM Deny Policies: Granted my user the roles/iam.denyReviewer at the Org level and confirmed no Deny Policies exist at Org/Folder/Project levels that would block run.services.invoke or target my user/service.
6. Organization Policies: Checked effective policies for the project:
   • constraints/run.allowedIngress: Confirmed this is Inactive / Inheriting, likely defaulting to allow-all traffic.
   • Reviewed other Org Policies; none seemed likely to block authenticated access.
7. VPC Service Controls: Confirmed via Cloud Console (Security -> VPC Service Controls) that the project is not part of any VPC-SC perimeter.
8. Test Container: Deployed Google's standard us-docker.pkg.dev/cloudrun/container/hello container to a new private service (test-hello-service), granted run.invoker to my user, and tested via curl with access token -> Still resulted in 401 Unauthorized. This rules out issues with my specific application/container.
9. Service Recreation: Completely deleted my-cloud-run-service using gcloud run services delete and redeployed it (privately); the 401 Unauthorized issue persists on the recreated service.
10. Identity Token Attempt: Briefly attempted using OIDC Identity Tokens, but ran into separate issues generating them (Invalid account Type error for user, NOT_FOUND / PERMISSION_DENIED issues when trying to set up SA impersonation, possibly due to inconsistent Cloud Build SA state).

Summary: Despite confirming the necessary run.invoker IAM role is present for my user, and ruling out standard blockers like IAP, Deny Policies, Org Policy ingress restrictions, and VPC-SC, authenticated requests using standard gcloud access tokens are consistently failing with 401 Unauthorized at the Cloud Run infrastructure level (logs confirm "request was not authenticated").

Question: Given this extensive troubleshooting, what other configurations, policies (perhaps less common Org Policies?), or potential environmental factors within Google Cloud could be causing this consistent 401 Unauthorizedresponse for authenticated users who demonstrably possess the run.invoker role? We do not have a paid support plan to open a direct technical case.

Any suggestions for further investigation would be extremely helpful!

Thank you.

[ageoff]

https://cloud.google.com/run/docs/authenticating/developers#curl - Make sure you are using the identity token and not the access token for Auth token:

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" $SERVICE_URL$

Note the "print-identity-token" instead of "print-access-token"

[birdma]

Thank you for the feedback, particularly the suggestion to use gcloud auth print-identity-token instead of print-access-token for the curl test, as Cloud Run expects an OIDC Identity Token. I agree this is the correct approach according to the documentation.

Unfortunately, I encountered specific errors trying to generate this token in my environment:

1. Directly as User: Running gcloud auth print-identity-token --audiences=[SERVICE_URL] resulted in ERROR: Invalid account Type for --audiences. Requires valid service account. My user account seems unable to mint audience-specific ID tokens directly.
2. Via Impersonation: I attempted to impersonate the default Cloud Build SA (...@cloudbuild.gserviceaccount.com) to generate the token. However, granting my user the required Service Account Token Creator role on that SA failed because the SA resource itself consistently returns a NOT_FOUNDerror (likely due to enforced Org Policies: cloudbuild.disableCreateDefaultServiceAccount and cloudbuild.useComputeServiceAccount). Cycling the Cloud Build API after temporarily disabling these policies did not resolve the SA NOT_FOUND issue.

So, while using an Identity Token is the goal, generating one is currently blocked.

The core issue remains: authenticated users (tested multiple accounts) with the run.invoker role consistently receive 401 Unauthorized when accessing private Cloud Run services (including Google's standard 'hello' container) via curlwith an access token. Browser access yields 403 Forbidden. Standard checks (IAM, IAP, Deny Policies, Org Policy Ingress, VPC-SC) haven't revealed the cause.

Does the inability to manage/find the default Cloud Build SA, or the enforced use of the Compute Engine SA for builds (due to Org Policy), suggest any other potential root causes or troubleshooting avenues for the persistent Cloud Run 401authentication failure?

Thanks again for any insights.

[ageoff]

Do you need the audience? Does it work without the audience? Agreed, the audience requires an SA, but if you cannot generate an SA could you confirm it works without the audience and then you know you just need a SA? Would confirm the root cause.