Dataplex DQ rules as code with Terraform

[Arstem]

Hi community, 

I have a question related to managing data quality rules as code with Terraform which is described here - https://cloud.google.com/dataplex/docs/manage-data-quality-rules-as-code#setting_up_your_github_repo...

Nevertheless, it works, my question is how to manage creation of several scans-since based on example and article we set only one table with rules for it. How can we define rules for several tables and deploy them?

[NorieRam]

Hi @Arstem,

Welcome to Google Cloud Community!

While Dataplex offers advanced use cases, you cannot define data quality rules for multiple tables within a single task as of the moment.

As @ms4446 mentioned about limitation:

Dataplex's Data Quality rules focus on single tables, with sql_expression designed for specific table contexts. While subqueries are supported, their use is limited to operations within the primary table of the Data Quality scan.

Here’s a workaround that you can do, create one YAML file per table containing its rules, then  use Terraform to deploy a separate google_dataplex_task for each YAML file, ensuring one task per table.

I hope the above information is helpful.

[Arstem]

Thank you for your response. But I believe there may have been a misunderstanding. My concern is about managing data quality rules as code using the Terraform-provided solution.

In the current implementation, it processes only one YAML file with rules for a specific table at a time. But what if I want to manage rules for multiple tables - say 10 YAML files? Ideally, I would like to run the solution and have it create 10 corresponding scans in Dataplex automatically.

[jvilchezv]

Hi @Arstem 

I have de similar quetion, did you find any domcument to help you?