Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
Forum Posts
Real-time Civic Media Dashboard on GCP – Feedback on High-Velocity Data Ingestion and Visualization
Hello Google Cloud Community,I’m Devanshu Dandekar, a Data Engineer working on a real-time civic dashboard for...
Dataproc cluster creation permission issue from Composer Env
I'm having a problem with creating dataproc clusters for my spark job using a DAG from the composer environmen...
Cloud composer 2 fails to check dataflow job status
Hi I need help with this issue.We recently migrated from Cloud Composer 1 to Cloud Composer 2 but we have issu...
Composer 3 - Monitoring frequently offline
Hello,I've been using Cloud Composer 3 for some time now. Recently I've noticed that my Environment Overview t...
schedule the upload of a local file to a GCP bucket
Hi,How can I create a pipeline to schedule the upload of a local file to a GCP bucket?I used this documentatio...
How to install python packages during runtime in dataproc serverless
In Dataproc, we have metadata option to install python packages in a dataproc cluster created through Composer...
OperationalError when using TriggerDagRunOperator
Hi, I'm newly setting up Cloud Composer on composer-3-airflow-2.10.2-build.12 and have encountered some errors...
Composer - multiple environments in dev tier
Hi all, We are currently using multiple composers instances when we need additional dev tiers(dev1, dev2 etc)....
Composer Airflow Dataset Error
Hi there! We met a scheduler error in Composer Airflow where scheduler does not work can continuously meet suc...
Composer pipeline to upload files from Google BigQuery to an S3 bucket
I have an s3 bucket (amazon) and I want to write a composer pipeline to upload csv files into the S3 bucket on...
Connect my mysql database with airflow composer
I have a airflow implementation that running in composer. I establish the database connection using airflow ui...
Airflow not able to parallelize tasks with .expand() - GCP Composer 3
Hello,I was developing dag pipelines in Airflow with Composer 3 set-up. Everything was working fine even with ...
Dataflow Temp Files Not Being Deleted
I have multiple Dataflow jobs that run on a daily basis and pick up data from GCS, apply transformations, and ...
At least one action must be selected for execution" in DataformCreateWorkflowInvocationOperator
Dear Google Cloud Community,I hope this message finds you well.I am encountering an issue while using the Data...
Composer workers are restarting frequently
@pgstorm148 I am using composer-2 [ composer-2.8.5-airflow-2.7.3 ] version in which i am running min= 2 schedu...
Can not Upgrade the composer Version 2
I have noticed from few days that I am not getting to UPGRADE Composer 2 image version. During my finding i fo...
No native solution SFTP to GCS? For real?
I am exploring options for copying/moving some files from SFTP to GCS, Google Cloud Storage.Does GCP for real ...
Running specific sqlx files using Airflow (Composer)
Hi, I want to execute specific Dataform (sqlx) files using Airflow (Composer).I know the best practice is to t...
Guidance on ETL Pipeline from Google Skill Boost to BigQuery
Hi All, I’m working on a proof of concept (PoC) to extract data from Google Skill Boost and load it into BigQu...
Composer 3 - Call Internal Cloud Function
Hello,I'm trying out Composer 3 to see if it fits my environment, and I've hit a problem when calling an inter...
Cloud Composer error with Dataflow Operator in deferrable mode
Hello!I'm currently testing my Composer 2 environment to upgrade from composer-2.9.11-airflow-2.10.2 to compos...
GKEStartPodOperator pods marked as failed despite succeeding
When deferring GKEStartPodOperator, sometimes the pod finishes successfully but airflow marks it as a failure,...
Cloud Composer - No Logs in Cloud Logging
Hello,I'm using Cloud Composer 2 and I've noticed that practically no logs seem to be flowing to Cloud Logging...
Restore Cloud Composer default connection
In Cloud Composer 2 (Version: 2.9.3+composer), I started getting a warning in several of my tasks `WARNING - C...
ConnectionTimeoutError inside Cloud Compsoser/Airflow when connecting to API
The API that I'm trying to connect is Iconik API which is a REST API and has a limit of 50 requests per second...
Cloud Composer 2 pypi package upgrade build timeout
So we are using Cloud Composer 2 version (composer-2.10.2-airflow-2.10.2) and want to upgrade the pre-installe...
Need Advice on Dynamic Dags in the Cloud Composer
Hey GCP community,We are using Cloud composer and using Dynamic Dag generation concept , but we storing the da...
Creating Composer 1 with Terraform fails
Hello. I'm getting this error when I'm trying to create a Composer 1 env with terraform.Error: Error waiting t...
How to really stop a DAG from running on startup
Hello,We use Airflow on a daily basis but everything runs in a few hours during the morning. After everything ...
Dataform ETLs too slow when run from Airflow
Hello,I'm having two problems when running Dataform ETLs from Airflow. For example, when running an ETL locall...