This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Log in to ask a question

Pub/Sub to GCS/BigQuery

Posted on 02-09-2024 06:54 AM
Nigal
Bronze 3
Bronze 3
Reply posted on --/--/---- --:-- AM

Hi ,

i am having a scenario like only when Data quality check is success i want to persist the data from pub/sub to GCS/Bigquery. I dont want the data which fails the DQ validation into GCS/BigQuery. Are there any GCP services available which will do this function. Please suggest

0 2 70
Topic Labels
0 Likes
2 REPLIES 2

Posted on --/--/---- --:-- AM
agi83
Silver 3
Silver 3
Reply posted on --/--/---- --:-- AM

Hi @Nigal 

Have a look at Dataplex https://cloud.google.com/dataplex?hl=en and these hands on labs https://www.cloudskillsboost.google/course_templates/681

 

0 Likes

Posted on --/--/---- --:-- AM
Nigal
Bronze 3
Bronze 3
In response to agi83
Reply posted on --/--/---- --:-- AM

Hi,

I had a look at these Dataplex templates but this dataplex wants data to be structured like a GCS/Bigquery data as an input and not directly from Pub/Sub. Actually i want to persist to GCS/Bigquery only if DQ validation is success. Please suggest.

0 Likes