Column Lineage in Google Dataplex

[hsenoussi]

Hello, 

I know that we can visualize Table lineage in Dataplex. Is it possible to visualize Column Lineage for BigQuery Table? 

Thanks in advance

[ms4446]

Dataplex now supports column lineage in preview. It's important to note that features in preview might be subject to change, and users should exercise caution when using them in production environments. To visualize column lineage in Dataplex, follow the steps in your previous response. However, be aware that column lineage is not yet supported for all data sources in Dataplex. Currently, column lineage is only available for BigQuery tables.

Data Lineage API: The Data Lineage API is also in preview and offers a programmatic way to access data lineage information specifically for BigQuery tables and columns. To utilize the Data Lineage API, you'll need to enable the API in your project and then employ a client library to make requests. For a deeper understanding and to get started, it might be beneficial to refer to the official Google Cloud documentation or any available quick start guides.

Third-party Data Lineage Visualization Tools: The third-party data lineage visualization tools you mentioned in your previous response all support column lineage for BigQuery tables. However, it's crucial to understand that not all of these tools may support the Data Lineage API. Therefore, it's advisable to check the documentation of each tool to determine if it's compatible with the Data Lineage API.

Recommendation: I strongly recommend verifying the capabilities of Dataplex and the Data Lineage API in the official Google Cloud documentation before finalizing your response. Additionally, it's a good practice to consult the documentation of any third-party data lineage visualization tools you're considering to ensure compatibility and support.

[hsenoussi]

Hello and thank you for your answer.

I still can't see how to access column lineage via dataplex UI. In my case I only want to visualize column lineage for bigQuery tables. So I figure out that won't block us. 
Could you please walk me through it ? where in the UI can I have the visualization ?

Thank you 

[ms4446]

How to Access Column Lineage in the Dataplex UI:

1. Navigate to the Dataplex page within the Google Cloud console.
2. Use the Search box to input the name of the BigQuery table for which you wish to visualize column lineage.
3. From the search results, select the desired table.
4. Once on the Entry details page, navigate to the Lineage tab.
5. The displayed lineage graph will showcase the lineage for all columns within the table.

Understanding the Lineage Graph:

• Nodes: Represent individual columns in the table.
• Edges: Symbolize the relationships between columns.
• Direction of Edges: Denotes the flow of data.

For instance, an edge like table1.column1 -> table2.column2 signifies that the column1 from table1 is utilized to populate the column2 in table2.

For a deeper understanding, you can click on any node within the graph to access more detailed information about the respective column.

Important: It's crucial to note that the column lineage feature in Dataplex is currently in its preview phase and is exclusively available for BigQuery tables.

[hsenoussi]

I am sorry but I can only see tables in my lineage graph. How could I access preview features in Dataplex ?

[injamed]

Hi! Would really like to up the question from hsenoussi - we don't see the option of column lineage either, though it was a highly waited feature.

Was the feature rolled out? Is it available for all BigQuery users?

Thanks.

[ms4446]

@All,

Apologies for any misunderstanding. The ColumnLevenLineage feature planned for Dataplex is still in its early stages of development and is not yet in private preview. Our product and Engineering teams are diligently working on it, and we'll ensure to keep you updated as soon as there are any advancements. Thank you for your patience and understanding.

[Sunder]

Hi,

In one of Next 24 presentation, it was highlighted that column level lineage is now in preview but i can't see this in UI yet. Can you please confirm is this feature is available yet?

Thanks,

Sunder

[ms4446]

The Column Level Lineage feature was released for private preview. Please reach out to your account team to be allow listed to use this feature.

[Sunder]

Thanks @ms4446 . I will reach out to account team.

Thanks,

Sunder

[emmanuelchacon]

Hi

I would like to know if the ColumnLevelLineage feature has had advancements, or if you know some dates release when we can use that

I appreciate the information about that