Any way to get pipeline metadata inside of the job in Vertex AI?

[jeongukjae]

Hi there,

I want to retrieve the Vertex AI pipelines' metadata including the run name in the jobs which are built using TFX and TensorFlow. The pipeline is launched in the Vertex AI Pipelines service, and each component is launched as a custom job in the "Vertex AI - Model Development - Training" service.

I searched the env vars exposed in the custom pipeline job and another way to retrieve it, but I couldn't find it.

Is it possible to access that information?

[lsolatorio · Accepted solution]

Hi @jeongukjae, 

Welcome and thank you for reaching out to our community.

A direct way to acquire pipeline metadata inside of a Vertex AI job is not available at the moment as these information are usually not exposed.

Consider exploring these resources as they relates to your use case.

• Introduction to Vertex ML Metadata
  • Track Vertex ML Metadata

• The TFX Components
  • TensorFlow Metadata (TFMD)
  • ML Metadata (MLMD)

Hope this helps.

[jeongukjae]

Thank you @lsolatorio! I didn't use the MLMD connection directly, but found that the Driver class in TFX can fetch the pipeline metadata (resolved from mlmd).