Cloud Armor - multiple rate limiters in a single Security Policy

[sokol8]

Hi everyone,

I would like to have several rate limiters that would operate on different time-horizons.

For example:

1) RateLimiter1 will allow a user (ip-address + path pair) make max 10 requests per minute and ban the user for an hour in case of threshold violation.

2) RateLimiter2 will allow a user (ip-address + path pair) make max 100 requests per 24 hours and ban the user for an hour in case of threshold violation.

I can add multiple Rate Limiters to Cloud Armor but it looks like I cannot make them work properly together.

Rate Limiter 1 will either DENY request or ALLOW request to pass through.

When RateLimiter1 make decision to ALLOW request to pass through, the request will never reach RateLimiter2 for evaluation.

My main question is: is it even possible to make Cloud Armor Rate Limiters work like that? Are other non-Google WAFs allow anything like that?

Thanks!

[jayeshmahajan]

not sure about cloud armor but aws waf allows multiple rate-based rules within the same WAF configuration, with custom thresholds and the ability to combine rules using and or conditions. Same goes for Cloudflare WAF: Offers Rate Limiting Rules with customizable thresholds, time windows, and responses, making it possible to implement multi-layered rate limits in a more seamless way.

[sokol8]

@jayeshmahajan  thanks for your response!

In CloudArmor I can add multiple Rate Limiting rules but it seems that only the topmost is being evaluated.

I cannot find a way how to get multiple Rate Limiting evaluated one after another.

I hope someone from GCP team can chime in on this.

[jamespatrickm]

Hi @sokol8,

Welcome to Google Cloud Community!

I’ve asked around and from what I could gather from my expert colleagues in Cloud Armor is that:

Cloud Armor rules are evaluated as either true or false. If true, the action prescribed by the rule will be taken and if false, it will proceed to evaluate the next rule. This means cascading rules is not possible and this includes rate limiting. Multiple rate limit rules in a policy will be evaluated separately. 

I have two options that could work for your use case:

1. Use some sort of hint i.e header or cookie that could indicate you have been here. It may be possible to use that as a trigger to exclude them from the first rate limit rule and they would fall to the second rate limit rule.
2. Consider using a rate-based ban. Set the throttle to the lower threshold and set the ban rate level to the higher value with the backoff of 60m. This won’t stop the traffic at the lower threshold, but it will constrict it and the user will be banned for the hour duration if there is continued abuse.

I hope this helps.

[sokol8]

@jamespatrickm  thanks a lot for the detailed response!

rate-based-ban is what we are already using. What we're missing is an ability to eveluate multiple rate limiting rules for the same incoming request.

I may be misunderstanding how Cloud Armor evaluates the rules. Your statement that "Multiple rate limit rules in a policy will be evaluated separately. " contradicts my experience

Here's what I see - I have the following rules:

1) Action: "Deny" , priority: 100, and a match: e.g "!request.path.contains('/api')"

2) Action: "Deny" , priority: 200, and a match: e.g "request.headers['user-agent'].contains('http-client')" 

3) Action: "Rate based ban", priority: 300 with ban-threshold of 10 requests per 1 minute

4) Action: "Rate based ban", priority: 400 with ban-threshold of 100 requests per 60 minutes

When an incoming requests is being analyzed against the list of rules it will have the following potential outcomes

1) enforcedSecurityPolicy.outcome="DENY" OR go to rule 2

2) enforcedSecurityPolicy.outcome="DENY" OR go to rule 3

3) enforcedSecurityPolicy.outcome="DENY" OR enforcedSecurityPolicy.outcome="ACCEPT" 

at this point the evaluation stops.

so Rule 4) with priority: 400 will never be evaluated.

To me it looks like we are not talking about cascading rules at all. It looks like the first Rate-based ban is a terminal rule. It doesn't matter how many more rules I will add with lower priority values - they will never get evaluated.

Is it possible to check my experience with your colleagues?

Maybe there's a single checkmark I need to set to get my subsequent rate-limiting rules evaluated?

Great thanks!

[sonslm]

Hi @sokol8,

I understand, that you would like to configure 2 different rate-limiting rule for the same request, correct?

Based on Google CA priority order matters as well. So, have you tried maybe to configure RateLimiter2 first, on priority 300, and then RateLimiter1, on priority 400? CA starts checking rule from top-to-bottom, which means, if you have a rule with 10 requests/ 1 minute "before" the 100 requests/ 24 hours, then the lower value will always evaluate and the higher never.
However, if we change the order, CA is going to check the higher threshold first, and if it is not a match, only then will it check the lower threshold.

Please let me know if this is something that can work on your end, or if you maybe found a solution for your use case in the meantime.
Best regards.

[sokol8]

Hello @sonslm - thanks for chiming in

I tried to configure two rate limiters with different rates - but the issue was that the first rate-limiter - whichever it was - was either rejecting traffic OR allowing it through. I cannot get any rule with higher prio to get evaluated AFTER my rate-limiter.

As I mentioned in my previous comments

>> Rate Limiter 1 will either DENY request or ALLOW request to pass through. <<

My tests showed this

1) enforcedSecurityPolicy.outcome="DENY" OR go to rule 2

2) enforcedSecurityPolicy.outcome="DENY" OR go to rule 3

3) enforcedSecurityPolicy.outcome="DENY" OR enforcedSecurityPolicy.outcome="ACCEPT" 

at this point the evaluation stops.

so Rule 4) with priority: 400 will never be evaluated.

Frankly I think that I may be missing something gross here, as everyone tells me my setup should be working while my logs analysis shows it doesn't.

[pranab16]

Hi, I am also facing exact same issue and Google support team mentioned the setup is correct. After escalating, they provided this documentation - https://cloud.google.com/armor/docs/rules-language-reference. This mentions lower rule will never be executed if request matches with higher rule (doesn't matter the outcome).

This shows immaturity with Google WAF and its architecture. They really need to work on getting parity with our Cloud Provider leaders like AWS, Cloudflare.