Cannot unpickle model from Model Registry when trying to make predictions

[dachlukenschieb]

I created a custom classification model (a scikit-learn RandomForestClassifier) using a CustomTrainingJob. The model trained successfully and appeared in the Model Registry in Vertex. The job used

container_uri = 'europe-docker.pkg.dev/vertex-ai/training/sklearn-cpu.1-0:latest'
model_serving_container_image_uri = 'europe-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-0:latest'

The model's model.joblib file was also successfully saved in the GCS bucket. When loading the model.joblib locally, I find that it contains the trained model. 

However, when I either try to deploy the model to an endpoint or create a batch prediction I get the error in the screenshot below. 

ERROR:root:Could not load the model: /tmp/model/0001/model.joblib. basic_string::resize.

The prior warning

Trying to unpickle estimator DecisionTreeClassifier from version 1.0.2 when using version 1.0

seems to suggest that, despite using Google's pre-built containers, the model is built using a different scikit-learn version (1.0.2) than the version that tries to unpickle the model file (1.0).

Does anyone have advice on how to fix this so that the model can be loaded/unpickled correctly by Vertex so that it can be used for prediction? 

Thanks a lot.