Google Cloud LB healthcheck logging not working in LOGS EXPLORER

[tacoma50]

Under COMPUTE ENGINE > HEALTH CHECKS I have a health check called https-healthcheck-443 and it shows LOGS=Enabled.
The health check is working OK with LB, because when I stop one of my two backend http servers and I look at LB's backend service, I see it shows 1 of 2 are Healthy as expected.  And when I start the stopped server, I then see 2 of 2 are healthy.

The problem is, I don't see any health check results in the LOGS EXPLORER with this query:
  logName="projects/project-prod/logs/compute.googleapis.com%2Fhealthchecks"

Furthermore, in the LOGS EXPLORER when I click on the ALL LOG NAMES pulldown and type in "compute" all I see is:
   COMPUTE ENGINE
      compute.googleapis.com/shielded_vm_integrity  

I have this same exact setup in another project called project-test and healthcheck logging returns results as expected with this query:
  logName="projects/project-test/logs/compute.googleapis.com%2Fhealthchecks"

The difference I notice here, is that in the LOGS EXPLORER when I click on the ALL LOG NAMES pulldown and type in "compute", I see:
   COMPUTE ENGINE
      compute.googleapis.com/shielded_vm_integrity
      compute.googleapis.com/healthchecks

Why is compute.googleapis.com/healthchecks available in my test project but not in my prod project?

[debolek]

If test works but prod doesn’t, most likely causes are :-

Health check logging isn’t actually enabled → Check with gcloud and update it.
IAM restrictions on Logging API → Ensure you have logging.viewer and compute.admin roles.
Log routing is different in production → Check gcloud logging sinks list.
Logging API isn’t enabled → Run gcloud services enable logging.googleapis.com.

[tacoma50]

Hi @debolek 

Thank you for the suggestions!   I have tried each of the four suggestions and I'm posting results here. Unfortunately, I still can't query the LB healthchecks in logs. 

The ONLY difference I am aware of  between prod and test projects is that prod uses zonal-dns and test uses global-dns(https://cloud.google.com/compute/docs/internal-dns)

1. Health check logging isn’t actually enabled → Check with gcloud and update it.
It looks like logging is enabled in both prod and test projects:

$ gcloud compute health-checks list --project myproject-prod
NAME REGION PROTOCOL
https-healthcheck-443 HTTPS

$ gcloud compute health-checks list --project myproject-test
NAME REGION PROTOCOL
https-healthcheck-443 HTTPS

$ gcloud compute health-checks describe https-healthcheck-443 --project myproject-test
httpsHealthCheck:
kind: compute#healthCheck
logConfig:
enable: true
name: https-healthcheck-443
selfLink: https://www.googleapis.com/compute/v1/projects/myproject-test/global/healthChecks/https-healthcheck-...

$ gcloud compute health-checks describe https-healthcheck-443 --project myproject-prod
httpsHealthCheck:
kind: compute#healthCheck
logConfig:
enable: true
name: https-healthcheck-443
selfLink: https://www.googleapis.com/compute/v1/projects/myproject-prod/global/healthChecks/https-healthcheck-...

2. IAM restrictions on Logging API → Ensure you have logging.viewer and compute.admin roles.
I added compute.admin role to my ID in my prod project and it did not help.
I did notice that in the TEST project I have a role of owner, so I added owner role to prod project and that did not help.

3. Log routing is different in production → Check gcloud logging sinks list.
I am not sure what to check here, but both prod and test look the same.
Is more sink data I need to collect?

$ gcloud logging sinks list --project myproject-test
NAME DESTINATION FILTER
_Required logging.googleapis.com/projects/myproject-test/locations/global/buckets/_Required LOG_ID("cloudaudit.googleapis.com/activity") OR LOG_ID("externalaudit.googleapis.com/activity") OR LOG_ID("cloudaudit.googleapis.com/system_event") OR LOG_ID("externalaudit.googleapis.com/system_event") OR LOG_ID("cloudaudit.googleapis.com/access_transparency") OR LOG_ID("externalaudit.googleapis.com/access_transparency")
_Default logging.googleapis.com/projects/myproject-test/locations/global/buckets/_Default NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT LOG_ID("externalaudit.googleapis.com/activity") AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT LOG_ID("externalaudit.googleapis.com/system_event") AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency")

$ gcloud logging sinks list --project myproject-prod
NAME DESTINATION FILTER
_Required logging.googleapis.com/projects/myproject-prod/locations/global/buckets/_Required LOG_ID("cloudaudit.googleapis.com/activity") OR LOG_ID("externalaudit.googleapis.com/activity") OR LOG_ID("cloudaudit.googleapis.com/system_event") OR LOG_ID("externalaudit.googleapis.com/system_event") OR LOG_ID("cloudaudit.googleapis.com/access_transparency") OR LOG_ID("externalaudit.googleapis.com/access_transparency")
_Default logging.googleapis.com/projects/myproject-prod/locations/global/buckets/_Default NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT LOG_ID("externalaudit.googleapis.com/activity") AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT LOG_ID("externalaudit.googleapis.com/system_event") AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency")

4. Logging API isn’t enabled → Run gcloud services enable logging.googleapis.com.
It looks like logging api is enabled in both prod and test:

$ gcloud services list --enabled --project myproject-prod |grep logging
logging.googleapis.com Cloud Logging API

$ gcloud services list --enabled --project myproject-test |grep logging
logging.googleapis.com Cloud Logging API

[tacoma50]

I'm  starting to think it has something to do with this...

When I am in TEST project and I go to LOGS EXPLORER and then in the ALL LOG NAMES pulldown I enter "compute", I see these two items appear under COMPUTE ENGINE:
healthchecks
shielded_vm_intgrity

But when I am in prod, I do not see those two items, all I see is shielded_vm_intgrity.
healthchecks, the one I need is missing.

Logs Explorer in TEST project: