Cloud Composer version upgrade - points to take note of?

[bdn77692]

Hi,

I am currently using a Cloud Composer instance with version: composer-2.4.1-airflow-2.5.3. As Composer 2.4.1 is no longer supported, I am considering an upgrade to the latest version: composer-2.9.2-airflow-2.9.1. Since it's a very large version jump, I would like to check with the community on key GCP-related factors I need to take note of before making the upgrade. What I have checked so far: 

• New version uses Python 3.11
• Airflow worker memory requirements in Python 3.11 (new composer version) are 10% higher compared to workers in Python 3.8 (old composer version)
• Worker concurrency is calculated using a different formula for newer Airflow versions
• Starting from GKE version 1.27.5, Cloud Composer environment clusters will start using SSD disks as persistent disks (how will this affect existing cluster setups)?

Will there be any changes to the GKE cluster or the cloud composer setup that I need to be aware of for the upgrade?

[mpinlac]

Hi @bdn77692 ,

Given the significant version jump, it's important to address several key factors to ensure a smooth transition. Here are some factors to consider upon upgrading: 

• Cloud Composer 2:

                - the environment's cluster scales automatically to meet the resource demands.

• Version deprecation and support:

                - Composer 2: Google supports Cloud Composer 2 versions for a period of time after the version release. During this period, Cloud Composer 2 environments that use these versions are fully supported.

• Airflow version Support:

                  -  Cloud Composer 2 supports two minor versions of Airflow 2.

                  - Airflow 2, Cloud Composer supports one patch version.

• Python Support:

                  - Cloud Composer supports Python 3: see versions for the major.minor.patch  of Python 3 specific Cloud Composer image.

• Cloud Composer versions and features:

                  - You can check the different features of each Composer version.

• GKE cluster:

                - Upgrading to a new Cloud Composer version may require or suggest upgrading your GKE cluster to a compatible version

                - You can also check on the release notes for the changes and new features in the latest GKE.

I hope the above information is helpful.