Multi table transformation cloud data fusion pipeline

[sheetal22]

Hi,

Is it possible in single cloud data fusion pipeline, to read multiple tables from one bigquery data set apply different transformations on each of the tables then write back to another bigquery dataset as individual tables.

For example:

Consider a dataset named 'data_in' in bigquery, which has 4 tables and we need to apply transformations on each table:

Table : Transformations

Table1 : Make column_1 uppercase

Table2 : concatenate two columns (like first_name + last_name  = full_name)

Table3 : create new column indicating avg(salary)

Table4 : Abbreviate state names

Load this transformed data to a bigquery dataset 'data_out' (different from input dataset) 

I am looking to create one pipeline to carry out above steps, my project requirement is that we don't want individual pipeline for each table but one pipeline should be able to read multiple tables and apply transformation on each table and load this transformed data to separate tables in Bigquery. 

So far, I tried using multiple database tables to read data and bigquery multi table to load data but have no idea how to do transformations in the same pipeline on individual table.

I tried using argument setter as well but it only reads one table even if table names are passed as array in JSON config file. 

Please suggest if and how is it possible. 

[ricconoel]

Hi,

Unfortunately reading from multiple BQ tables and at the same time transforming data in a single pipeline is not yet possible. There is an existing feature request for this but there is no guarantee on the ETA of when this feature will be implemented. You can "star" the issue as the engineering team considers an issue with a lot of stars.

[sheetal22]

Hi,

Thank you for your response. I see that the request you mentioned was created a year ago, don't know if it is actually going to be implemented. Is there any possibility apart from star the issue to escalate it?