This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Error when trying to interact with google aiplatform related to protobuf library

Posted on 01-18-2024 08:16 AM
PhilippMCH
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

I'm experiencing an error when trying to launch a run via the Vertex AI API from python:

Training script copied to:
gs://mma440_projects/aiplatform-2024-01-18-16:41:59.494-aiplatform_custom_trainer_script-0.1.tar.gz.
Traceback (most recent call last):
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
return self._descriptor(**value)
TypeError: bad argument type for built-in operation

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
return self._descriptor(**value)
TypeError: bad argument type for built-in operation

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/rules/message.py", line 36, in to_proto
return self._descriptor(**value)
TypeError: bad argument type for built-in operation

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/philipp/project/vertex-ai-bug-encaps/launch_vertex_job.py", line 218, in <module>
args.func(args)
File "/home/philipp/project/vertex-ai-bug-encaps/launch_vertex_job.py", line 70, in launch_script_job
job = aiplatform.CustomJob.from_local_script(
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/google/cloud/aiplatform/jobs.py", line 1619, in from_local_script
job = cls(
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/google/cloud/aiplatform/jobs.py", line 1298, in __init__
job_spec=gca_custom_job_compat.CustomJobSpec(
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/message.py", line 581, in __init__
pb_value = marshal.to_proto(pb_type, value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 211, in to_proto
return type(value)(self.to_proto(proto_type, i) for i in value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 211, in <genexpr>
return type(value)(self.to_proto(proto_type, i) for i in value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 228, in to_proto
pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/rules/message.py", line 41, in to_proto
return self._wrapper(value)._pb
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/message.py", line 581, in __init__
pb_value = marshal.to_proto(pb_type, value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 228, in to_proto
pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/rules/message.py", line 41, in to_proto
return self._wrapper(value)._pb
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/message.py", line 581, in __init__
pb_value = marshal.to_proto(pb_type, value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 211, in to_proto
return type(value)(self.to_proto(proto_type, i) for i in value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 211, in <genexpr>
return type(value)(self.to_proto(proto_type, i) for i in value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 228, in to_proto
pb_value = self.get_rule(proto_type=proto_type).to_proto(value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/rules/message.py", line 41, in to_proto
return self._wrapper(value)._pb
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/message.py", line 581, in __init__
pb_value = marshal.to_proto(pb_type, value)
File "/home/philipp/project/vertex-ai-bug-encaps/venv/lib/python3.10/site-packages/proto/marshal/marshal.py", line 222, in to_proto
proto_type.DESCRIPTOR.has_options
AttributeError: 'ProtoType' object has no attribute 'DESCRIPTOR'

I have provided a minimal working example in https://github.com/PhilippMisofCH/vertex-ai-bug-encaps with all the details. Interestingly, a colleague and I could not reproduce the error on two other machines with the same python and python-package versions. To further encapsulate the problem I also tried running it inside a apptainer container as described in the link. The biggest difference I see right now is that I run it on Arch Linux whereas the other machines were using Ubuntu 22.04 and Rocky Linux 8.8.

Let me know if you need further information.

Thanks in advance for your help!

A colleague and I have tried to reproduce the error 

Edit: Corrected an url

Solved Solved
1 2 2,691
Topic Labels
Jump to Solution
1 ACCEPTED SOLUTION

Posted on --/--/---- --:-- AM
PhilippMCH
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

So as far as I understand the issue was that in the aiplatform.CustomJob.from_local_script() function one of the passed environment_variables={...} was not a (non-empty) string but an empty dictionary. Passing an empty string still resulted in an error but a non-empty string succeeds as expected. For the future it would be nice if the error is realized at a higher level than somewhere inside the protobuf library.

View solution in original post

Jump to Solution
0 Likes
2 REPLIES 2

Posted on --/--/---- --:-- AM
PhilippMCH
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

So as far as I understand the issue was that in the aiplatform.CustomJob.from_local_script() function one of the passed environment_variables={...} was not a (non-empty) string but an empty dictionary. Passing an empty string still resulted in an error but a non-empty string succeeds as expected. For the future it would be nice if the error is realized at a higher level than somewhere inside the protobuf library.

Jump to Solution
0 Likes

Posted on --/--/---- --:-- AM
Poala_Tenorio
Staff
Reply posted on --/--/---- --:-- AM

Make sure that all the packages, including the google-cloud-aiplatform library and its dependencies, are up to date.

 

Show More
pip install --upgrade google-cloud-aiplatform

 

Verify that you and your colleague are using the same version of Python. Also, check the version of the protobuf package installed in your virtual environment.

 

Show More
pip show protobuf

 

If the version is not up to date, upgrade it using:

 

Show More
pip install --upgrade protobuf

 

As you mentioned, you are running the code on Arch Linux, while your colleagues are using Ubuntu and Rocky Linux. There might be subtle differences in system configurations that are affecting the behavior. Ensure that your Arch Linux machine has all the required dependencies installed.

Since you've tried running the code inside a container, make sure the container environment is set up correctly. The container should include all the necessary dependencies and libraries.

Jump to Solution