This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Data form pricing

Posted Sunday
dipucs
New Member
Reply posted on --/--/---- --:-- AM

Hi,

I am exploring using dataform to execute sql workflows. After testing I have noticed costs attributed to dataplex. Does dataform uses dataplex internally. Is there any option to disable that?. Thank you

best regards

Dipu Chemmala Sadanandan

0 1 28
Topic Labels
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
ruthseki
Staff
Reply posted on --/--/---- --:-- AM

Hi dipucs,

Welcome to Google Cloud Community!

Yes, the managed Dataform service on Google Cloud is automatically integrated with Dataplex to provide enhanced data governance.

The costs you are observing are almost certainly from the Data Lineage API, which is a part of the Dataplex service family. After every successful run, Dataform automatically analyzes your SQL to determine column-level dependencies and sends this information to the Data Lineage API. This populates the lineage graphs you see in the BigQuery and Dataplex UIs. While basic metadata cataloging is very cheap, the processing of these lineage events is what incurs specific Dataplex charges.

You can disable this integration and its associated costs. The most direct and recommended way is to disable the Data Lineage API for your Google Cloud project. You can do this by navigating to "APIs & Services" > "Enabled APIs & services" in the Cloud Console, finding the "Data Lineage API," and clicking "Disable."

However be mindful of the consequences of disabling:

  • Your Dataform SQL workflows will continue to execute perfectly.
  • You will stop incurring Dataplex lineage costs.
  • You will lose the automatic data lineage feature. The lineage graphs for tables created by Dataform will no longer be generated, making impact and root-cause analysis more difficult.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

0 Likes