Start an executor from a notebook that has dependencies on other files

[ReneWooller]

We have a notebook A.ipynb which imports a module B.py.

Running A.ipynb from the Vertex AI Workbench works fine - however when we submit A.ipynb to Executor, it only copies the A.ipynb file without the dependency.

We ended up copying and pasting the contents of the supporting file into the notebook, but it's not very good code reuse. It's probably possible to use gsutil to manually copy files up to a bucket and then download them when the script executes, however those copied files won't change when the original file changes in the git repository. So really it would be more ideal for the depenancy to be able to be specified when the file is submitted to the executor. 

Is there some way to enable this? What do other people do who have files that are shared dependencies between notebooks?

[ruthseki]

Hi @ReneWooller,

Welcome to Google Cloud Community!

Here are some workarounds that you may try in managing dependencies in Vertex AI Workbench:

You can package your dependencies into a Python package and install it in your notebook environment. This way, you can update the package in your git repository, and the changes will be reflected when you install the package in your notebook.

• Create a setup.py: Define your module B.py as a Python package. This allows you to install it easily using pip. You may refer to this link for guidance on packaging your code as a pip module.

Another option is to create a `requirements.txt` file that lists all your dependencies. When you submit your notebook to the executor, you can include a step to install the dependencies from this file.

You may also try to use Vertex AI Workbench's built-in features. Vertex AI Workbench has features to manage dependencies and environments. You can create custom environments with all your dependencies and use them in your notebooks. Check out the Vertex AI Workbench documentation for more details.

With regard to what you mentioned about `gsutil` to copy files to a bucket and download them when the script executes, you can ensure the files are always up-to-date by automating this process using a CI/CD pipeline that syncs your git repository with the cloud storage.

You can also use GitHub integration. Vertex AI Workbench supports native GitHub integration, which allows you to pull the latest changes from your repository directly into your notebook environment. This can help keep your dependencies up-to-date.

You may also check this Vertex AI Workbench tutorials for more detailed guidance which you may find useful.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.