Cloud Run Job <> Service

[n-dua-observe]

Hi there,

I created a Cloud Run Job along with a Cloud run service that is accessible over internet (no VPC, auth)

Both were deployed on us-central1. 

However, the cloud run job fails to reach the cloud service (simple HTTP load test that pings an endpoint) as the service always returns 404 errors.  Running the same container locally (or in GCP terminal) works just fine.

The current workaround I found is changing the job's region to us-east1 for it to reach the service properly. The service deployed is just this (https://github.com/GoogleCloudPlatform/avocano)

Does anyone know if this is a bug or a limitation?

Thanks!

Nikhil  

. 

[snef]

Nickil,

have you seen the log of job after first run and error?

Could you share with us?

[n-dua-observe]

Yes! I'm running a k6 job against some service endpoints. Added debugging flags and you can see the 404 errors:

Link for the erroneous run: https://cloudlogging.app.goo.gl/Szw4WNufQ1rT8U3S9

[snef]

I can't see this log because i'm blocked from IAM.

Copy and past hear a snippet of error

[n-dua-observe]

Sure.

Here's the request:

{
  "textPayload": "time=\"2023-04-07T20:27:30Z\" level=info msg=\"Request:\\nGET /api/products HTTP/1.1\\nHost: content-eng-nikhil.firebaseapp.com\\nUser-Agent: k6/0.43.1 (https://k6.io/)\\nAccept-Encoding: gzip\\n\\n\\n\" group= iter=1016 request_id=f7cd81c5-3a5e-414b-61fe-de05fd356b96 scenario=view_random_product source=http-debug vu=1",
  "insertId": "64307cb200088da322fc0765",
  "resource": {
    "type": "cloud_run_job",
    "labels": {
      "project_id": "content-eng-nikhil",
      "job_name": "load-test-2",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-04-07T20:27:30.560547Z",
  "labels": {
    "run.googleapis.com/task_attempt": "0",
    "run.googleapis.com/task_index": "0",
    "instanceId": "00121789b33b353feb73f46e1c9c5f693d929e1a7adb4bd904669e2330748f935be3e6b20e7adf1fff5fbb17b9abbf0b0e6b77a871749555671cdfc1620be8423d26",
    "run.googleapis.com/execution_name": "load-test-2-v28cq"
  },
  "logName": "projects/content-eng-nikhil/logs/run.googleapis.com%2Fstderr",
  "receiveTimestamp": "2023-04-07T20:27:30.625499837Z"
}

The response (in us-east1 this is a 301/200, below is 404 for us-central1):

{
  "textPayload": "time=\"2023-04-07T20:27:30Z\" level=info msg=\"Response:\\nHTTP/2.0 404 Not Found\\nContent-Length: 10601\\nAccept-Ranges: bytes\\nAlt-Svc: h3=\\\":443\\\";ma=86400,h3-29=\\\":443\\\";ma=86400,h3-27=\\\":443\\\";ma=86400\\nCache-Control: max-age=0\\nContent-Type: text/html; charset=utf-8\\nCross-Origin-Opener-Policy-Report-Only: same-origin; report-to=\\\"gfe-default_product_name\\\"\\nDate: Fri, 07 Apr 2023 20:27:30 GMT\\nReport-To: {\\\"group\\\":\\\"gfe-default_product_name\\\",\\\"max_age\\\":2592000,\\\"endpoints\\\":[{\\\"url\\\":\\\"https://csp.withgoogle.com/csp/report-to/default_product_name\\\"}]}\\nStrict-Transport-Security: max-age=31556926; includeSubDomains; preload\\nVary: x-fh-requested-host, accept-encoding\\nX-Cache: HIT\\nX-Cache-Hits: 18324\\nX-Served-By: cache-chi-kigq8000098-CHI\\nX-Timer: S1680899251.565496,VS0,VE0\\n\\n\\n\" group= iter=1016 request_id=f7cd81c5-3a5e-414b-61fe-de05fd356b96 scenario=view_random_product source=http-debug vu=1",
  "insertId": "64307cb20008bcdf5a00ea18",
  "resource": {
    "type": "cloud_run_job",
    "labels": {
      "job_name": "load-test-2",
      "project_id": "content-eng-nikhil",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-04-07T20:27:30.572639Z",
  "labels": {
    "run.googleapis.com/task_index": "0",
    "instanceId": "00121789b33b353feb73f46e1c9c5f693d929e1a7adb4bd904669e2330748f935be3e6b20e7adf1fff5fbb17b9abbf0b0e6b77a871749555671cdfc1620be8423d26",
    "run.googleapis.com/task_attempt": "0",
    "run.googleapis.com/execution_name": "load-test-2-v28cq"
  },
  "logName": "projects/content-eng-nikhil/logs/run.googleapis.com%2Fstderr",
  "receiveTimestamp": "2023-04-07T20:27:30.625499837Z"
}

[knet]

Hi Nikhil, is this still happening? My first thought is that maybe the egress settings on the two jobs are configured differently, could that be the issue?

[n-dua-observe]

Hi! Yes it's still present.

Egress was checked and the two jobs were setup to be identical indeed