This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Batch Job + Composer (Airflow) + Python

Posted on 03-23-2023 10:48 AM
Jack-Michaelson
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hi all,

I'm trying to set up an environment for the Cloud Composer. The goal is to have airflow tasks start batch jobs, and wait for them to finish. Unfortunately, google.cloud.batch_v1 (version 0.9.0) depends on proto-plus==1.22.0, while composer-2.1.10-airflow-2.4.3 needs proto-plus==1.19.6 . So I cannot update the environment to add the needed package. Does anyone know a work-around?

0 2 521
Topic Labels
0 Likes
2 REPLIES 2

Posted on --/--/---- --:-- AM
Joevanie
Staff
Reply posted on --/--/---- --:-- AM

Google Cloud has a new Batch service that processes batch jobs. You may want to check it out as it may be helpful for your use case.

0 Likes

Posted on --/--/---- --:-- AM
Jack-Michaelson
Bronze 1
Bronze 1
In response to Joevanie
Reply posted on --/--/---- --:-- AM

Thank you Joevanie. That's exactly the service I'm asking about. It has a python API that I'd like to use in conjunction with Cloud Composer. But its python package depends on proto-plus==1.22.0, while the Cloud Composer environment needs proto-plus==1.19.6 . So currently I don't know how to use the two together. My question was whether there is any way around this issue. If there is no work-around now, does anyone know whether future releases will solve this issue? And if so, how soon?

0 Likes
Top Labels in this Space
Top Solution Authors
View all