Triggering multiple requests when editing a pipeline in Dataflow

[luizalves]

I've been getting the following message when creating/editing a new Dataflow pipeline in GCP:

Quota exceeded for quota metric 'Job template requests' and limit 'Job template requests per minute per user' of service 'dataflow.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'.

I believe it may be a problem on the platform because when looking at Chrome's DevTools I identify multiple requests in a few seconds with the response code 429, as shown in the attached image.

I hope this can be resolved soon, thanks in advance!

[ms4446]

This message is indicating that the rate of job template requests to the Dataflow service from your project has exceeded the quota limit set by Google Cloud Platform (GCP). The HTTP 429 response code stands for "Too Many Requests" and is returned when the user has sent too many requests in a given amount of time.

Quotas are set to prevent abuse of the service and to ensure fair usage among all consumers of the service. They can be set at various levels, such as per project, per user, or per service.

The specific quota that is being exceeded here is 'Job Template Requests Per Minute Per User'. This means that the number of job template requests made per minute by a single user has exceeded the set limit.

Here are a few steps you can take:

1. Check your quota usage in the GCP Console: Navigate to the IAM & Admin page, and then to Quotas. Filter by the dataflow.googleapis.com service to view your usage. This should help confirm if you are indeed hitting the quota limit.

2. Throttle your requests: If you're making requests programmatically, consider adding some delay between requests to avoid hitting the rate limit.

3. Increase your quota: If your usage is legitimate and you need to make more requests than currently allowed, you can request a quota increase from the GCP Console. Navigate to the IAM & Admin page, then to Quotas, filter by the dataflow.googleapis.com service, select the quota you want to increase, and click "Edit Quotas" to submit your request. Google will review your request and may increase your quota based on your usage patterns and needs.

Remember that not all quota increases are granted, as Google has to maintain service availability for all users. Also, quota increases can take some time to be approved and become active, so they're not an immediate solution to quota issues.

[luizalves]

Hello! I understand about the issue of quotas and I checked the limits in
IAM and none are exceeded. They are all normal, including
dataflow.googleapis.com.
The problem in question that I pointed out is on the GCP pipeline page,
when editing, the platform is triggering many requests per minute and I
believe this is the reason for the problem of exceeded quotas. You can
observe this in the print I put in the previous comment.

--
[image: #] <>

*Luiz Alves | * Engenheiro de dados de back-end
*E: * (PII Removed by Staff)
* W: * (PII Removed by Staff)
[image: fb] <> [image: eu]
<> [image: dois]
<>

[ms4446]

I see, this sounds like a more specific issue with Google Cloud Dataflow's UI causing an unusually high number of requests, leading to the quota being exceeded.

Here are a few suggestions on how to handle this:

1. Try Another Browser or Incognito Mode: Sometimes extensions or browser-specific issues could cause problems with web applications. Try using another browser or opening a new incognito window and see if the issue persists.

2. Contact Google Cloud Support: If you have a support contract, this issue sounds like it would be appropriate to escalate to them, particularly if you can provide the screenshot and other details you mentioned.

3. Google Cloud Issue Tracker: You could also report this issue on Google Cloud's public issue tracker. Be sure to include as much detail as possible, including the screenshot and details of your use case.