Dataflow HttpError 429 when requesting to trigger a dataflow template

[bharath-03]

Hi all,

   We created a dataflow template and running dataflow pipelines from it since a month or two. Today we received a weird error related to it which is "HttpError 429 when requesting <url> returned Internal service error: Failed to configure shuffle service for job <job_id>". But the next time we ran it triggered the pipeline and ran fine. I was going through the source of this error but unfortunately i couldn't find any. Can some one help me know what is this about.

  Job details : It is a batch dataflow pipeline. Source BQ query, Sink Pub/Sub.

Thank you in advance

[josegutierrez]

It may be caused due to a CPU usage because it is recommended to have 80 - 90% CPU usage for the workers to configure the shuffle service, and with the 429 error It is shown because it passed the limits, so my guess was that the usage was being used in another process the first time you tried it and it finished when you launched it the second time.