Hello,
I am trying to execute one Dataflow job only within a DAG on Composer. The Dataflow job runs successfully when I look at the Dataflow job screen, but in Airflow it shows failed once the job starts to tear resources down. The error seems to indicate it is looking for job status but can't find it. The job status link seems to be looking in us-central1 even though Composer and everything runs in us-east1, but not sure if that is the issue. I clicked on the link provided for projects.locations.jobs.get and it indeed is a 404 error. Has anyone ever encountered this issue?
<HttpError 404 when requesting https://dataflow.googleapis.com/v1b3/projects/strongsville-city-schools/locations/us-central1/jobs/2... returned "(6daa40014a0306c4): Information about job 2022-08-23_17_54_49-15122819176641177679 could not be found in our system. Please double check that the API being used is projects.locations.jobs.get (https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/get). If the API being used is projects.locations.jobs.get, please double check the id (2022-08-23_17_54_49-15122819176641177679) is correct. If it is please contact customer support.". Details: "(6daa40014a0306c4): Information about job 2022-08-23_17_54_49-15122819176641177679 could not be found in our system. Please double check that the API being used is projects.locations.jobs.get (https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/get). If the API being used is projects.locations.jobs.get, please double check the id (2022-08-23_17_54_49-15122819176641177679) is correct. If it is please contact customer support.">
LinkedIn
Twitter
