This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Cloudrun issue "Error 403 (Forbidden)!!" when sending many request at once

Posted on 03-14-2024 08:32 AM
bfircef
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hello everyone,

I'm facing an issue with a cloud run service. It works fine, I send request and receive responses but when I try sending a lot of requests almost 40/s it fails with this message received by the client :

 

<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">\n  <title>Error 403 (Forbidden)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n  </style>\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n  <p><b>403.</b> <ins>That’s an error.</ins>\n  <p>Your client does not have permission to get URL <code>/api/decrypt</code> from this server.  <ins>That’s all we know.</ins>\n

 

 The message doesn't appear into the cloud run service logs. I only have it as response.

My maximum number of concurrent requests that can reach each instance is at 1000 and the number of request sent is aproximately 40-45.

Do you have an idea of what could be my issue?

P.S: As I said beofre, I can make requests so it's not really about service account permissions or whatever. It's only when I send many requests at simultanously that I get that answer.

Thank you in advance.
Best regards,

0 3 2,046
Topic Labels
0 Likes
3 REPLIES 3

Posted on --/--/---- --:-- AM
christianpaula
Staff
Reply posted on --/--/---- --:-- AM

Hi @bfircef,

Welcome to Google Cloud Community!

You're encountering a throttling issue, not a permission problem, because:

  • Regular requests work fine, eliminating permission issues.
  • Error originates before Cloud Run logs, suggesting a load balancer or CDN block.

Here's how to diagnose:

  1. Check Cloud Run resource utilization in Cloud Monitoring for spikes during request bursts.
  2. Investigate load balancer/CDN logs for throttling events around the burst time.
  3. Review Cloud Run's concurrency settings and adjust if needed.
  4. Consider enabling Cloud Run autoscaling for dynamic traffic handling.
0 Likes

Posted on --/--/---- --:-- AM
bfircef
Bronze 1
Bronze 1
In response to christianpaula
Reply posted on --/--/---- --:-- AM

Thanks for you quick response @christianpaula !

  1. In the default graphs of cloud run I have the Max concurrent requests at 150 less than the limit 1000 
    bfircef_0-1710521805990.png
  2. In terms of network, it's pretty simple, I only have a VPC but no custom load balancer configured. The ingress control is on "All". The authentication is on "Allow unauthenticated invocations". I'm no sure how to check throttling here.
  3. I reviewed again and it's 1000, a number not reached
  4. I have Autoscaling params as Min instances: 2, Max instances: 100

    Do you have any other hint for me please?
0 Likes

Posted on --/--/---- --:-- AM
christianpaula
Staff
Reply posted on --/--/---- --:-- AM

We can rule out Cloud Run throttling because:

  • Concurrency limits (1000) are high enough.
  • Resource usage wasn't maxed out during the burst.

Let's investigate elsewhere:

  1. Cloud Monitoring: Check for network errors, high response times, or resource spikes during the burst.
  2. Network Bottleneck: Use cloud provider tools (e.g., Network Intelligence Center) to investigate network issues.
  3. Application-Level Throttling: Review your Cloud Run service code for any built-in throttling mechanisms.
  4. Client-Side Throttling: Check your client-side code for rate limiting configurations.

Additional Tips:

  • Gradual Increase Test: Send requests gradually increasing volume to pinpoint the bottleneck.
  • Burst Logging: Add detailed logging in your service during bursts to capture specific errors.
  • Consider Cloud CDN: Explore Cloud CDN for caching and load balancing to potentially prevent throttling.
0 Likes