How can we get Google to be ethical over daily limits?

[mark4]

I've had a web site on GAE for many years and set a daily limit of $5. This was hit maybe a couple of times a year due to people misconfiguring their server to download without stopping.

On Friday 2nd Feb 2024 at 11:30PM a misconfigured server started downloading. I was asleep and when I looked next morning at 9:10AM discovered 29,000 downloads amounting to 85GB had taken place in that time. I blocked the IP using firewall rules at 9:15AM

Normally my download cost ("out bandwidth") is between $0.10 and $0.20 per day, so for the 2nd and 3rd Feb I expected to pay at most $0.40. However, given the excess downloads, had the limit been in place (which I'd set at $5), my total cost would have been $10.

But without the daily limit, GAE kept serving. Due to time zones differences, for me the charging day is from 8AM to 8AM. So my bill for 2nd Feb is $74.33 (11:30PM on the 2nd to 8AM on the 3rd); and my bill for 3rd Feb (8AM to 9:15AM when I stopped the downloads) is $10.16, so $84.49 in total.

As GAE's helpful billing report says, "Feb 1 - 5, 2024 (total cost) $84.66 ↑ 10,234.39%".

So clearly GAE knows that excessive downloads were happening. Nor would it be rocket science for GAE to recognize this as a mistake or DoS attack (same download of big file repeated 100s then 1000s of times from same IP). So clearly Google has the technical expertise to be able to reinstate the daily limit—or at least to automatically prevent this kind of excess download.

Incidentally, the buget alert emails are useless: after the event and no use overnight. And as for the so called programmatically disabling, well, that's beyond my web programming abilities. (I've asked a company who've helped me before to give me a quote for doing this.)

I can't be the only small scale GAE user who relied on the daily limit. Nor the only one tied into GAE and not really able to migrate. So for me and those in a similar position, Google is a monopoly. They know perfectly well that people relied on that daily limit, yet disabled it.

I wish there was some way to get Google to do the ethical thing and restore the daily limit—or at the least to offer automated excess download protection that would simply stop serving any IP after it did more than a given number of downloads in any one 24 hour period.

[NoCommandLine]

- To programmatically shut down your App, see this blog article from us with step by step guide and sample code. It might end up not being as difficult as you thought it would be.

- Something else that might help is to set the billing alerts for much lower percentages than 100 e.g have billing alerts for 50%, 75%, 90% of your daily budget. There's no guarantee that you'll see them (like what you just experienced). You're simply trying to increase your chances of not being caught flat-footed. Hopefully, Google is able to find a way to restore 'spending limits' to GCP.

... NoCommandLine ......

 https://nocommandline.com

A GUI for Google App Engine
    & Datastore Emulator

[daskalou]

Are you able to put your files into something like Backblaze B2?

It has an S3 compatible API and lowest cost to store, retrieve and serve your data from any provider.

[mark4]

I've asked a company that's done web work for me before to quote for a solution. The outline is that I'll store my files in a bucket and when users click to download the URL will redirect to a 'limiter' program that will track IP x timestamp x filename and providing there aren't too many downloads from the same IP in a given timeframe, the download will 'just work'; otherwise, it'll error.

I still think it is a disgrace that Google dropped daily limits.

(Also I find this forum very buggy: I often get errors when I try to Post.)

[micdearmas]

Yes, it's kind of buggy. More often than not, I have to open a second tab in the browser and choose to discard whatever was auto-saved (after I typed my reply in the original tab) then paste my reply and post right away 😰