GC uptime monitoring causing 250k 404 errors on a site per month

[ross20]

Hi All,

I'm CTO for a small'ish Australian website agency.

It's just come to our attention that approx 250,000 404 errors are being generated per month in the log files for one of the websites that we host and support.

We can see in the logs that:

1. The user agent is GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring)
2. We're getting many hits per minute from the above agent.
3. Most if not all of these hits are for malformed URLs, hence all the 404 errors.

We have used Google Cloud uptime monitoring ourselves, but in this instance, we're fairly sure that this is not an uptime monitor that we've set up. We've asked the customer to check themselves.

But if we can't find the source, what can we do? Is there any way/service that Google provides that can find the owner of the account that is hammering/spamming our website with malformed URL requests?

Do we just go ahead and block in the firewall all the Google IPs that are associated with these requests?

fyi: the offending IPs include:

104.197.30.241
35.185.178.105
35.185.252.44
35.186.159.51
35.186.167.85
35.186.176.31
35.187.114.193
35.192.92.84
35.195.128.75
35.197.32.224
35.198.194.122
35.198.224.104
35.198.248.66
35.198.36.209
35.198.39.162
35.199.12.162
35.199.123.150
35.199.126.168
35.199.77.186
35.199.90.14
35.203.129.73
35.205.205.242
35.205.234.10
35.205.72.231
35.221.55.249
35.224.249.156
35.233.165.146
35.233.167.246
35.233.206.171
35.236.207.68
35.236.221.2
35.238.118.210
35.238.3.7
35.239.194.85
35.240.124.58
35.240.151.105

Of the ones I've tested, these IPs all resolve to googleusercontent.com

Thanks all,

Ross

[kolban]

Howdy Ross,

While you investigate on your end, I'd tempted to suggest filling out this form:

https://support.google.com/domains/contact/abuse

If you are receiving unsolicited/undesired Uptime checks from an unknown party, then it is possible that there is a misconfiguration on their part ... but either way, this might be a good start.

[ross20]

Thanks Kolban. I've filled out that form.

Looking forward to getting to the bottom of this!

[ross20]

Any ideas anyone?

We're still no closer to working out how we can find the source Google Cloud Uptime monitoring account that is performing this (malformed URL) uptime monitoring, which is in effect a DDoS on attack on our site.

We assume the malformed URLs in the uptime monitoring are accidental, rather than malicious, but we just don't know.

[ad7]

Hi ross20 - Did you get a response to your abuse report? Also, were you able to check with your customer if this was intentional? IMO GCP will unlikely be able to share customer/account details of the customer as that would be breach of the customer's privacy.  Thanks!

[ross20]

I've checked with the customer and they can't yet find if it's a problem with one of the checks they've setup.

I did get a response from my abuse report, but it redirected me to another form to fill out, where I was forced to choose the best available options that were very different from what I was actually trying to reporting, so I don't know what sort of response I'll get from this second attempt.

Thanks for responding ad7.

[ross20]

Still no progress on this.

We're recording in the region of 10k hits per day from Google Cloud Uptime monitoring.

I guess I just start manually blocking IP addresses...

[anilmoregfs]

Have you found the root cause of this issue? We are also facing the same problem—thousands of uptime checks. We have cloud armor to block it, but from the errors, it looks like a misconfiguration somewhere.

[ad7]

Hi anilmoregfs - Can you pls share more details? Do you have any projects with uptimes configured targeting the URI where you are seeing these failures? What errors you are seeing?

[anilmoregfs]

I opened a case with Google support, and they provided me with the list of all GCP projects having uptime checks. We removed the uptime checks from those projects and the issue was resolved.

[ross20]

Hi @anilmoregfs - I'm *not* comfortable that I got to the root cause of this. Fortunately for us the 404 hits decreased a lot, and have remained low since:

March '23 - 267,455
April '23 - 151,379
May '23 - 1,295
June '23 - 702

It's possible that the client themselves set this monitoring up in an incorrect way, but they've not admitted to this if it was them.