This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Cloud composer data size

Posted on 09-23-2024 02:20 AM
pavan26
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hi,

Is there a way to check the total amount of data that is being loaded in GCP big query via cloud composer (Airflow environment) on a daily basis

0 1 106
Topic Labels
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
caryna
Staff
Reply posted on --/--/---- --:-- AM

Hi @pavan26,

Welcome to the Google Cloud Community!

Here are some possible solutions for monitoring data ingestion in BigQuery:

  • INFORMATION_SCHEMA.JOBS: Use INFORMATION_SCHEMA.JOBS to find the total bytes processed for a specific job and get insights into your BigQuery data loading activities. You can query job_type, total_bytes_processed, and select additional columns according to your requirements, following the provided schema.
  • Cloud Monitoring: Leverage Cloud Monitoring to track key BigQuery metrics, including the amount of data and the number of rows uploaded to a specific table. Monitoring these metrics can serve as a proxy for tracking data usage during load jobs.

For more details, refer to the BigQuery monitoring documentation.

I hope the above information is helpful.

0 Likes
Top Labels in this Space
Top Solution Authors
View all