External function in Kubeflow Component.

[Im_Monjil]

How to call an external python function inside a kubeflow component and run the VertexAI Pipeline job?

[ruthseki]

Hi @Im_Monjil,

Welcome to Google Cloud Community!

You want to use your existing Python code (in external files) inside a Kubeflow pipeline running on Vertex AI. To do this, you need to structure your project into:

1. External Functions: Your reusable code lives in separate Python files (like utility functions).
2. Kubeflow Component:
• You create a Python file that defines a Kubeflow component using the kfp.dsl.component decorator or kfp.components.create_component_from_func.
• Inside this component, you import and call your external functions.
• You clearly define the inputs and outputs of this component.
3. Pipeline Definition:
• You create another file that uses kfp.dsl.pipeline to define the entire pipeline.
• You include your component within the pipeline structure.
• You compile the pipeline into a JSON file.
4. Pipeline Execution:
• You create a script that uses Vertex AI Python client to:
• Upload the compiled pipeline.
• Create and run the pipeline job on Vertex AI.

Here are the key steps:

• Encapsulation: The component acts as a wrapper, making your external code usable in the pipeline.
• Dependency Management: You'll need to ensure any libraries used by external functions are available within the component's execution environment (typically, via the base_image and packages_to_install).
• Clear Interfaces: Define clear input and output types for the components for data flow within the pipeline.
• Compilation: You compile the pipeline description into a JSON file, which is then submitted to Vertex AI.
• Vertex AI Execution: The Vertex AI Pipelines service takes the compiled pipeline and executes the components in a controlled environment.

I suggest checking these links to Kubeflow Pipelines and Vertex AI Pipelines documentation, specifically on components, pipelines and creating/running pipelines on Vertex AI:

• Vertex AI Pipelines
• Build a pipeline
• Run a pipeline
• Kubeflow Pipelines: A Step-by-Step Guide
• Unleashing the Power of Kubeflow: A Guide to Training Machine Learning Pipelines

Also here are some tutorials that can guide you through creating, compiling, and running a Vertex AI pipeline using components:

• Intro to Vertex Pipelines
• Vertex AI Pipelines: Pipelines introduction for KFP
• How to use the Kubeflow Pipelines (KFP) SDK to build Vertex AI Pipelines that use lightweight Python...

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.