Hi all,
We created a dataflow template and running dataflow pipelines from it since a month or two. Today we received a weird error related to it which is "HttpError 429 when requesting <url> returned Internal service error: Failed to configure shuffle service for job <job_id>". But the next time we ran it triggered the pipeline and ran fine. I was going through the source of this error but unfortunately i couldn't find any. Can some one help me know what is this about.
Job details : It is a batch dataflow pipeline. Source BQ query, Sink Pub/Sub.
Thank you in advance