Composer - multiple environments in dev tier

[crazyrevol]

Hi all, 

We are currently using multiple composers instances when we need additional dev tiers(dev1, dev2 etc). We deploy all of our DAG folder from repo into the storage bucket used by composer. 

I want to be able to deploy multiple branches of repo to the same storage bucket( but in different folders - dev1, dev2 etc) used by one composer only(dev composer). In this scenario - how can I have multiple environments in the same composer to be able to run the dag with same dag_id at the same time ? May be a way to separate context by namespace or something else ?

DOes such a solution exist? 

[dawnberdan]

Hi @crazyrevol,

Welcome to Google Cloud Community!

You can simulate multiple dev environments within a single Cloud Composer instance using logical separation and naming strategies. While Composer doesn't support namespaces or true isolation, the following techniques can help:

1. Use Subdirectories in GCS for Environment Separation: Organize DAG code by environment (e.g., dags/dev1/, dags/dev2/). Ensure that each DAG file defines a unique dag_id (e.g., my_dag_dev1) to avoid conflicts.

2. Parameterize DAGs with Environment Context: Use environment variables, Airflow Variables, or JSON config files to inject environment-specific settings into your DAGs. Combine this with unique dag_ids for each environment to run them in parallel. Check this document as it explains how to override Airflow configuration options, which might be useful for customizing your setup.

3. Isolate Workloads Using Task Queues: Assign different task queues (e.g., dev1_queue, dev2_queue) to route task execution via Celery or KubernetesExecutor. This provides resource isolation but doesn't replace the need for unique dag_ids.

4. Dynamic DAG Generation via Factory Pattern: Use a common template to programmatically generate multiple DAGs with unique IDs. This scales well across multiple environments and keeps code DRY.

5. Environment-Specific Deployment Pipelines: In your CI/CD process, deploy only the relevant DAGs per environment or use symlinks/folder filters to control what's loaded. While true DAG ID isolation isn’t possible within one Composer instance, these strategies help simulate multi-env behavior without maintaining separate Composer environments.

You can check this case to explore different strategies for handling multiple environments within a single Composer instance and this blogpost that details the strategies for running DAGs or tasks in different clusters.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.