context timeouts during composer upgrades and python package additions

[tt_davisj]

Show More

We are experiencing issues with cloud composer upgrades and the addition of pypy packages to the system.  Both represent with terminal messages of "ERROR: context deadline exceeded".  It appears that the timeout is 35mins.

This error occurs during upgrade:

composer-2.3.5-airflow-2.4.3 --> composer-2.8.5-airflow-2.7.3

We are able to complete an. upgrade by removing all custom packages from the module.  However, we receive the timeout again when the attempt is made to add the packages back.

The pypi packages in question are:

cloud-sql-python-connector --> [pymysql]>=0.8.1

We would like to find a resolution to this error however the timeout appears to be coming from cloud build and does not appear to be adjustable.  

JD

[jangemmar]

Hi @tt_davisj,

Welcome to Google Cloud Community!

The “context deadline exceeded” error usually means that an operation is taking longer than the allowed time limit to complete. It appears that the 35 minute timeout is likely imposed by Cloud Build, which Cloud Composer uses behind the scenes for environment updates.

The most common problem with PyPI packages in Cloud Composer are dependency conflicts. Also, a large number of custom packages in your Composer environment can add overhead to the upgrade process.

To isolate the issue, here’s the workaround that you can do:

• You can try to install a different version of the PyPI package to test.
• Increase the machine type or resources allocated to Cloud Build for Composer operations. This might involve configuring the machineType parameter in your deployment configuration.
• You can also try to optimize network configurations for better throughput and lower latency.

I hope the above information is helpful.