Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Repository could not be reached, Dataform connected to Gitlab

Hi

I have implement data transformation pipeline on Google Cloud Composer, by using

1. "DataformCreateCompilationResultOperator" to create comilation on Dataform
2. "DataformCreateWorkFlowInvocationOperator" to create invocation to run models on Dataform 

which process by order 1 >> 2


The error happen when "DataformCreateCompilationResultOperator" try to create compilation on Dataform which connected to Gitlab repository. It raised this error 

 

google.api_core.exceptions.InvalidArgument: 400 Remote repository 'xxx.git' could not be reached.

 

Chanan_0-1692762860862.png

After that I add number of retries to 2 (1 min interval) to DAG config, it was  retried and it can be proceed but it not stable, sometimes they need 5 retries.

Chanan_1-1692763925319.png

ref : Google's troubleshooting link

After that I try to find solution on Google's troubleshooting, the google inform that "There is no need to take any action. Unless GitHub or GitLab issues persist, the subsequent scheduled releases can be successful"

The question is, what is the cause of this error and how to solved it, with appropriate action,  the "retry" method is not stable, maybe the error will happen more often in the future.

I'm really troubled by this issue because the customers are really worry

Can you please give me the solution, Thank you very much

 

0 2 763
2 REPLIES 2