Estimating Dataform pipeline scheduling costs

[BarboraS]

Hi there,

I am trying to unpack the pricing structure for Composer & Workflows tied to Dataform scheduling. The cost for workflows consists of internal and external steps. What constitutes an "internal step" and an "external step" in this context?

Is every executed SQLX a separate internal step? Every time it executes (or fails)? What exrternal steps can there be for running a DF project?

Thanks!

B.

[Joevanie · Accepted solution]

According to Google Cloud’s documentation, Dataform pricing depends on the resources that you use to execute your pipelines, such as Cloud Composer, Cloud Scheduler, and Cloud Workflows. A step is a unit of work performed by a workflow. An internal step is a step that calls another workflow or performs an operation within the same project. An external step is a step that calls an external service or API. In the context of Dataform pipelines, an internal step could be a SQLX script that transforms data within BigQuery. An external step could be a call to Dataform’s API to run or test a pipeline.

[BarboraS]

Great to hear from you and confirming my understanding of the docs.