Error while creating model valuations using ModelEvaluationClassificationOp

[trk]

Hello GCP Community,

I am encountering an issue while attempting to access data in the `ModelEvaluationClassificationOp` which is a pre-built gcp component. The error message I am receiving is:

"LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist."

Here are the details of the relevant code snippet:

`ModelEvaluationClassificationOp` expects two GCS input paths (`ground_truth_gcs_source` and `predictions_gcs_source`), but I am encountering difficulties accessing the specified buckets.
ground_truth_gcs_source: list[str] = ["gs://trk-ai-models/sklearn-income-pred-model/income_classification_ml_metadata/input_predcition_data/gcs_groundtruth.jsonl"]
predictions_gcs_source: dsl.Input[system.Artifact] = "gs://trk-ai-models/sklearn-income-pred-model/income_classification_ml_metadata/output_predictions_data/prediction-test_income_model-2024_03_19T06_00_04_089Z"

I have verified that the specified buckets and respective files in Cloud Storage exist and are accessible.

 reference snippet : https://github.com/kubeflow/pipelines/blob/master/components/google-cloud/google_cloud_pipeline_comp...

please provide guidance on how to resolve this issue or suggest troubleshooting steps?

Thank you in advance for your assistance.

[nceniza]

I would suggest to file a support case for this as engineers have a better visibility of your project and your project's resource usage that will help them to investigate the issue you are having. 

https://cloud.google.com/contact

[trk]

Can I contact through the basic support plan, or do I need to have a standard support plan? If not, are there any alternatives to raise a support ticket without any cost?

[jsndai]

Hi,

---

"LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist."

---

This warning message might be misleading and unrelated to the real error that caused your component to fail. It was fixed in the newer versions of the `google-cloud-pipeline-components` Python package. I recommend upgrading this package to the latest version `2.14.0` and import the `evaluation_automl_tabular_classification_pipeline` from there.

Thanks,
Jason Dai

[asiapit00]

Hi Jason / All,

Thanks for the information in this thread. I'm experiencing the exact same "LookUpInode: input/output error, Error in iterating through objects: storage: bucket doesn't exist." error when running a Vertex AI model evaluation pipeline for tabular classification in the europe-west3 region.

My situation seems slightly different as I am creating and launching the pipeline run directly through the Vertex AI Console UI, not by compiling a Python script with the SDKs.

Looking at the failed evaluation step details in my UI-generated runs, I can confirm it's using the container image gcr.io/ml-pipeline/model-evaluation:v0.9.4.

Based on your previous answer, it sounds like this version might contain the issue causing this misleading error. While upgrading the google-cloud-pipeline-components Python package fixes it for SDK users, this doesn't seem to apply when initiating the run from the standard UI components, as the run still uses the v0.9.4 image.

(I also confirmed basic GCS connectivity and permissions are okay by running a simple gsutil ls custom job test from the same region/SA, which succeeded.)

Is there a plan to update the standard evaluation components used by the Vertex AI Console UI to a version that includes the fix? Or is there a recommended workaround for users encountering this specific error when creating evaluation runs through the UI?

Thanks for any insights!