View time series metrics on a PipelineJob?

[yohannpitrey]

I'm fairly new to Vertex pipelines but I've been working with Kubeflow for a while. I have a rather complex pipeline with several kfp components, which I've recently copied over to Vertex. The pipeline is running well, I'm able to start a `PipelineJob` and then submit it with `job.submit(experiment='my-exp')`.

So far I've been able to log simple metrics using `metrics: Output[Metric]` in the component's signature and `metrics.log_metric({name: value})` in the component's body. Now I'd like to view some time series metrics but I can't work out how to do this starting from a `PipelineJob`. All the examples I've seen start by starting an experiment run using `aiplatform.start_run("run-1")` and then running `aiplatform.log_metrics({"metric1": [...values...], "metric2": [...values...]})`. Is there a way to achieve the same thing starting from a `PipelineJob`? Shouldn't a component be able to write a time series as a Metric?

[ruthseki · Accepted solution]

Hi @yohannpitrey,

Welcome to Google Cloud Community!

You can't directly log time series metrics to Vertex AI from a Kubeflow Pipelines component in the same way you would using the aiplatform client library's log_metrics function with a manually started run. To log time series metrics, Vertex AI Experiments requires a backing Vertex AI TensorBoard instance. The Kubeflow Pipelines metrics.log_metric function only supports logging single scalar values at a single point in time, not time series data.

The aiplatform library's approach of using start_run and log_metrics is designed for managing runs independently of Kubeflow Pipelines. Kubeflow Pipelines handles its own metrics logging through its own system, which integrates with the Vertex AI monitoring UI but doesn't directly expose the same functionality.

To understand the differences between Vertex AI Pipelines and Kubeflow Pipelines, see Migrate from Kubeflow Pipelines to Vertex AI Pipelines.

Moreover, to achieve a similar effect of viewing time series data related to your PipelineJob in Vertex AI, you need to adjust your approach:

1. Modify Your Components to Log Data Frequently:

Instead of trying to directly log time series, log individual data points frequently within your components. Each call to metrics.log_metric will create a separate data point in the pipeline's logs. The frequency depends on how granular your time series needs to be. For example, if you're tracking training loss, log it after each epoch.

2. Visualize in Vertex AI Monitoring:

After the PipelineJob completes, you'll see these individual metric points in the Vertex AI monitoring UI. While not a true time series in the sense of a continuous line, you'll see a sequence of points representing the metric's value over time. The timestamps will be implicit based on the execution time of each log_metric call within your component. You can then use the Vertex AI UI's chart capabilities to visualize the data as a line chart, giving a time series representation.

3. Custom Logging:

For greater control, log your data to a file (e.g., CSV or JSON) within your component and then upload the file to Cloud Storage after the component finishes. You can then use the BigQuery integration in Vertex AI or other visualization tools to analyze this data as a time series. This gives you more flexibility in how you manage and visualize the time-series data outside the inherent limitations of the Kubeflow Pipelines metric logging.

In summary, while direct time series logging isn't directly supported through the metrics object in Kubeflow Pipelines components, you can achieve a very similar result by logging frequent data points and then using the Vertex AI monitoring UI or custom storage and visualization solutions. 

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[yohannpitrey]

Thank you for taking the time to answer! This is going to save me lots of time scratching my head. I had a look at the link you shared regarding migrating to Vertex AI pipelines and I can report that I'd already gone through that process. I won't be able to use simple runs as my pipeline involves multiple components with complex dependencies between them and requires some components to run in parallel.

Could you please expand on what you mean by this:

> "You can then use the Vertex AI UI's chart capabilities to visualize the data as a line chart, giving a time series representation."

What specifically do you mean by "Vertex AI UI's chart capabilities"? Ultimately I'd like to view some metrics on a chart and ideally I would want to do this from the Run details page. Is this what you are refering to? Is there a way to view the metrics there?