I am currently using Vertex AI's Multimodal Embedding Model (https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings)
I was able to get the Image and Text examples running just fine using the Python SDK, however, when I try to run the following Video example (from the above link), I get the following error
import vertexai
from vertexai.vision_models import MultiModalEmbeddingModel, Video
project_id = 'my-multimodal-search-project'
location = 'us-central1'
vertexai.init(project=project_id, location=location)
# Document metadata
video_path = 'gs://multi-modal-assets/videos/dog_jumping_short.mp4'
description = 'dogs playing on a field'
model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
video = Video.load_from_file(video_path)
embeddings = model.get_embeddings(
video=video
)
# Video Embeddings are segmented based on the video_segment_config.
print("Video Embeddings:")
for video_embedding in embeddings.video_embeddings:
print(
f"Video Segment: {video_embedding.start_offset_sec} - {video_embedding.end_offset_sec}"
)
print(f"Embedding: {video_embedding.embedding}")
print(f"Text Embedding: {embeddings.text_embedding}")
The error I get when running this sample code is:
Traceback (most recent call last):
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
return callable_(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Video embedding failed with the following error: Deadline"
debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.81.234:443 {grpc_message:"Video embedding failed with the following error: Deadline", grpc_status:16, created_time:"2024-05-23T15:53:15.429704-04:00"}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/jacky.l/Code/multi-modal-test/test.py", line 18, in <module>
embeddings = model.get_embeddings(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/vertexai/vision_models/_vision_models.py", line 1190, in get_embeddings
response = self._endpoint.predict(
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/cloud/aiplatform/models.py", line 1614, in predict
prediction_response = self._prediction_client.predict(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py", line 851, in predict
response = rpc(
^^^^
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jacky.l/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.Unauthenticated: 401 Video embedding failed with the following error: Deadline
The specific error is "Deadline", which implies timeout. I tried to figure out how to extend the timeout from 60 seconds to something higher, but it does not seem the code exposes any parameters to change the timeout.
Any ideas how I can fix this?
LinkedIn
Twitter
