Vertex AI Pipeline job fails due to binary incompatibility (numpy, pandas)

[gabrielmc]

Hi!

I was working with this notebook sample on Colab Enterprise: Vertex AI RLHF Notebook Sample on GitHub and the Vertex AI pipeline job fails when the training step starts.

The pipeline logs display an error that looks like a mismatch between numpy and pandas versions. That's the error:

ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval from pandas._libs.interval import Interval

But I seem to have the correct versions installed: numpy 1.26.4 and pandas 2.2.2. As stated in pandas' doc the versions should be compatible: https://pandas.pydata.org/pandas-docs/stable/whatsnew/v2.2.2.html

What you expected to happen:
The pipeline job to complete

Steps to reproduce:
-Open the notebook  (Vertex AI RLHF Notebook Sample on GitHub)
-Fill the input fields
-Run cells up to 'Run the pipeline using Vertex AI Pipelines'
-Wait while the pipeline runs (6 - 8 minutes)
-Check Vertex AI UI and logs for error

I have checked other forums about this issue (pandas failure in flex app engine deployment ), but in all of them this issue was cause by incompatible versions.

Uptade: The issue happens when running the notebook without modifying it (changing dataset and params) from GitHub and when modified as well.

Any help would be appreciated!

[nikacalupas]

Hi gabrielmc,

Welcome to the Google Cloud Community!

The error "ValueError: numpy.dtype size changed, may indicate binary incompatibility" is typically caused by mismatched binary builds of these packages. Here are the potential solution to resolve the issue:

• Check if there's a way to clear the cached container images or dependencies in Vertex AI Pipelines, as doing so can force the pipeline to use the latest versions or rebuild containers with your updated dependency configurations.
• Inside the notebook, before running the pipeline, explicitly reinstall NumPy and pandas. This ensures you're using the exact versions you specified: 
  • !pip uninstall -y numpy pandas
  • !pip install numpy==1.26.4 pandas==2.2.2]

• If you're using a custom Docker image for your pipeline, rebuild the container image after reinstalling NumPy and pandas to ensure the correct versions are included in the container.
• Alternatively, consider using slightly older versions that are still compatible:
  • !pip install numpy==1.25.0 pandas==2.1.3

• Ensure your input data isn't causing issues by checking for missing values (NaNs), unusually large data types (e.g., int64, float64 if not necessary), and incorrect file encoding (ensure it's UTF-8).
• Increase the logging within your pipeline components to better identify the source of the error, and focus on any specific error messages related to NumPy or pandas that may provide further insight.

You may also check this documentation for more information about Vertex AI Pipelines.

If the issue persists, you may reach out to Google Cloud Support. When reaching out, include detailed information and relevant screenshots of the errors you’ve encountered. This will assist them in diagnosing and resolving your issue more efficiently.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[gabrielmc]

Thanks a lot for the reply.

I tried that and it didn't work, unfortunately. 

I decided to run the sample notebook as it is in Vertex AI's GitHub (with the provided sample datasets) in a new runtime instance and I got the same error. So the issue is not dataset related and it's likely not something that I did wrong.

It would be helpful to know if other developers are having the same issue running this notebook.