Vertex AI Pipelines adoption and alternatives

[Jbbqqf]

I am trying to understand the adoption of the "pipelines" feature of Vertex AI.

According to a small sample size around me, more than half of the people using Vertex AI use the pipelines feature, which is a good sign.

The biggest criticisms are that:

• it takes time to get used to it / to setup the first pipeline, but it's worth it
• the feature adds delay when ML Engineers need a fast feedback loop
• it's overkill for small use cases

I have two questions:

1. Would you say those criticisms are fair?
2. What are the alternatives, especially for small use cases? Standalone cloud run jobs?

[dawnberdan · Accepted solution]

Hi @Jbbqqf,

Welcome to Google Cloud Community!

Vertex AI Pipelines is a powerful tool, but it does come with a learning curve. It might not be the best fit for every situation, so it's crucial to weigh whether the benefits justify the added complexity.

Fairness of the Criticisms:

• Adjustment Period: It’s common to experience a learning curve with any new tool, especially one with a structured approach like Pipelines. It requires adapting to a new way of organizing and managing your ML workflows.
• Delays in Feedback Loops: This is a valid concern, particularly for exploratory work or rapid prototyping. Pipelines introduce an extra layer of abstraction and potential overhead, which can slow down quick iterations.
• Excessive for Smaller Applications: Whether this applies depends on what you consider “small.” If you’re working with a single model and minimal preprocessing, Pipelines might seem excessive. However, even for smaller projects, the ability to track experiments and manage dependencies can be valuable

Alternatives for Smaller Use Cases:

• Cloud Run: Suitable for straightforward tasks, especially if you don’t need the orchestration and complex dependencies that Pipelines provide. Cloud Run is quick to set up and handles basic tasks like model training and prediction.

• Kubeflow Pipelines: If you're familiar with Kubernetes and need more flexibility than Vertex AI Pipelines offers, Kubeflow Pipelines provides similar functionality with more control.
• MLflow: A popular open-source platform for experiment tracking and model management. It’s a good option if you want more control over your ML workflows and are comfortable with Python.

Consider using Vertex AI Pipelines when:

• You have complex workflows with multiple steps. Pipelines are great for organizing and managing workflows with various dependencies.
• You need version control and experiment tracking. Pipelines offer features that support reproducibility and collaboration
• You want to automate your ML workflows. Pipelines can automate repetitive tasks like data preparation, model training, and deployment.
• Scalability and reliability are important. Pipelines are designed to be scalable and reliable, making them well-suited for production-level applications

I hope the above information is helpful.

[Jbbqqf]

Thank you for your valuable feedback. It will help me give context to my colleagues.