Platform Block?: Cloud Run Service Returns Google 404 Despite Being Healthy and Publicly Configured

[moggy8]

I've been troubleshooting this extensively with the help of AI (including Gemini) and after multiple rounds of debugging I cannot find the solution. I'm working on a small, single-person, project so I don't have any organizational policies created that would impede the health, and believe there is a platform level impediment for the service. Thank you in advance for any help you can provide -  I'm newer to the Cloud Run services. Let me know what supplemental information I can provide.

Project ID: appthree-86b53

Service Name: sports-ingest-fanout

Region: us-central1

Problem Description: Our Cloud Run service, sports-ingest-fanout, is unreachable from the public internet and returns a generic, Google-branded 404 Not Found error page. This occurs even for explicitly defined health check endpoints. However, the service is verifiably healthy, running, and functions correctly for internal traffic.

Comprehensive Diagnostics Performed:

We have undertaken an exhaustive debugging process and can confirm the issue is not with the application code, its dependencies, or its direct configuration.

1. Application-Level Fixes (Completed): We initially identified and fixed several application-level bugs. The application is now stable:

• Corrected Startup: Fixed the package.json to use node index.js instead of the incorrect functions-framework.

• Added Server Listener: Added the required app.listen(process.env.PORT || 8080) to index.js to ensure the server starts correctly.

• Created Dependencies: Added the necessary gcloud tasks queues create command to our deployment script to ensure the required Cloud Tasks queue exists before the service starts.

• Current Status: Logs definitively show the container starts, the server listens on port 8080, and the service passes all startup probes. The application is not crashing.

2. Environmental and Networking Checks (Completed): We have ruled out all common user-configurable causes for this issue:

• Public Access Confirmed: The service is deployed with the --allow-unauthenticated flag, and gcloud run services get-iam-policy confirms the allUsers principal has the roles/run.invoker role.

• No VPC Connector: The service is not attached to a Serverless VPC Access connector.

• No Load Balancer: There are no Global External HTTPS Load Balancers configured in the project that could be interfering.

• Internal Traffic Succeeds: A backfill job initiated from within our GCP environment runs successfully. It correctly hits the /ingest endpoint, creates tasks, and the worker processes them. This proves the entire application logic is sound.

The "Smoking Gun" Evidence: The core issue is the discrepancy between internal and external traffic.

• Internal Request (e.g., backfill job): Succeeds.

• External Request (e.g., curl <service-url>/healthz): Fails with a Google-branded 404.

Conclusion & Request: The fact that a healthy, running container is reachable from internal GCP services but not from the public internet is definitive proof of an environmental policy blocking public ingress. The request is being intercepted by Google's infrastructure before it reaches our container.

We request that your team investigate the project-level and platform-level configurations that are not visible to us. Specifically, please investigate:

1. VPC Service Controls: Is project appthree-86b53 part of a service perimeter that restricts run.googleapis.com without a corresponding public ingress rule?

2. Organization Policies: Is there an effective Organization Policy on this project, such as constraints/run.allowedIngress, that is overriding our service's public configuration?

We have exhausted all possible application-level and user-configurable fixes. Please assist us in identifying the platform-level block.

[Joy_S]

Hi @moggy8,,

Welcome to Google Cloud Community!

Here are a few things to double-check and some possible causes:

• Verify if the service URL is correct by running: 

  gcloud run services describe sports-ingest-fanout --region=us-central1 --format="value(status.url)"

• App Isn’t Ready at Startup: Sometimes, if the container starts but your app isn’t yet listening on the port, Cloud Run may return 404 during cold starts. You can fix this by adding a startup probe to delay traffic until the app is fully ready.

• Ingress Settings Blocking Public Access: If your Cloud Run service is set to allow only internal traffic (like from GCP or a load balancer), public requests will return a 404. Check this with:

  gcloud run services describe sports-ingest-fanout --region=us-central1 --format="value(spec.template.metadata.annotations.run.googleapis.com/ingress)"

     If it’s not set to all, you can fix it with:

gcloud run services update sports-ingest-fanout --region=us-central1 --ingress=all

• VPC Service Controls (VPC-SC) Blocking Access: If your project is part of a VPC Service Controls perimeter, public access to run.googleapis.com can be blocked based on IP or project identity—even if your service allows public traffic. You can look for VPC-SC policy violations in Cloud Logging using this filter: 

  resource.type="audited_resource"
  
  log_name="projects/[PROJECT_ID]/logs/cloudaudit.googleapis.com%2Fpolicy"
  
  resource.labels.method="run.googleapis.com/HttpIngress"

• Default run.app URL Disabled: Cloud Run services can be configured to disable the default run.app domain. When this is the case, any public request to that URL will return a 404—even if the service itself is healthy. You can check for this by looking for this annotation in your service YAML: 

  run.googleapis.com/default-url-disabled: true

           If the default URL is disabled, you’ll need to access the service through a custom domain or re-enable the default.

• 404 Coming from Google Infrastructure: If none of the above apply, the 404 may be coming from Google’s internal routing layer (e.g., GFE or RDS), not your app. This can happen if:
  • The routing config didn’t update properly
  • The service revision wasn’t resolved
  • There’s a temporary issue in the control plane

These kinds of 404s don’t show up in Cloud Logging, since the request never reaches your app.

If you’d like to dig deeper, you can add this header to your request to get debug information: 

X-Return-Encrypted-Headers: request_and_response

This will return extra routing info in the response, which GCP support can help decode.

If the workaround above doesn't work, you can contact Google Cloud Support for a more in-depth analysis. When contacting them, please provide comprehensive details and include screenshots. This will help them better understand and address your issue.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.