Private Uptime Check VPC_REQUEST_TIMEOUT

[ktan_08]

Hi,

To summarize:
I am trying to configure an uptime checks for ApigeeX Instances

as per on this workaround documentation: 

https://cloud.google.com/apigee/docs/api-platform/system-administration/health-check-mig-workaround#... 

one of the step it provided says to create a PSC Connected Endpoints 

base on this, what I thought of doing was instead of doing the whole work around documentation replacing my PSC NEG to MIG, what if I created a uptime private checks instead for this private service connect connected endpoints. (Note: I already do the whole workaround MIG configuration and everything is working. What I would like to see if there is something that I can do to not create the MIG and change the backends as our only goal is to have a healtchecks for each instance)

Now, I am trying to configure Private Uptime Checks as specified in this documentation: https://cloud.google.com/monitoring/uptime-checks/private-checks#gcloud-cli_1

This is what my configuration looks like:

but as per testing:

I am getting "VPC_REQUEST_TIMEOUT"

What I would like to know was:

1. Is there any wrong in my configuration ?
2. My idea of doing this kind of uptime checks for ApigeeX instance is incorrect ?
   

Tagging

@dchiesa1 @kurtkanaskie 

Thank you.

[williamssean]

So I think you are asking for the following:

• Apigee Northbound networking with PSC
• Utilize a Health Check
• avoid MIGs

Your screenshot is using Uptime Checks which are for Synthetic Monitoring and you can't use that for a Google Cloud Global Application Load Balancer.  Synthetic Monitoring is for a different use case.

Per our docs - failover with multiple PSC NEGs with active health checks are not supported (see the second bullet point).  However, it does say that you can use outlier detection on the PSC NEG backend.  

So when you configure your Global external Application Load Balancer (not the classic ALB) and you setup a PSC NEG as a backend, then if you scroll down you will see the Advanced configurations section.  Check the box Enable under Outlier detection.  It looks like you can also edit those settings, if necessary.  This outlier detection will evict unhealthy backends based on how you configured it.

[ktan_08]

Hi @williamssean 

We already have outlier detection enabled.

Our goal is to have some sort of notification to us to let us know that specific instance is down in case of failure as we currently have 3 instance configured (asia, eu, us) and the PSC Negs doesn't support health checks we have no way of knowing it.

As for the uptime checks, I am not using it against Global Application Load Balancer I am using it against PSC connected endpoints which is the same as stated in this doc for configuring private uptime checks (https://cloud.google.com/monitoring/uptime-checks/private-checks#gcloud-cli_1) 

If you have any recommendation without using private uptime checks just to have our goal achieve(mention above) that will be great.

Thank you.

[kurtkanaskie]

Hmmm, interesting idea, but...

Even if you manage to configure private uptime checks, or even public uptime checks, what would you do when a check failed?

I suppose, you could set an alert / notification, then use Event Arc to trigger a function to remove the unhealthy backend. I suppose you could also use the notification when the event cleared to re-add the backend.

I don't think that would provide the necessary response time for an unhealthy backend to prevent the client from seeing backend errors. That BTW is one of the drawbacks of using Outlier Detection, it relies on actual error responses from the backend which the client will also see.

[ktan_08]

Thanks @kurtkanaskie for the insights.

as for now, we are going to rely on Outlier Detection for the removal of unhealthy backends (we will going to observe and see).

Our goal for now, is to have someway to alert/notify us if specific instance is down (we have 3 asia, eu, us). We have PSC Negs setup and have no way to find out if for example 1 of our 3 instance is down.

If you have any recommendations to achieve our goal that will be helpful. Thank you.