Maximum Local Video File Size for Embeddings

[picheny]

I have video files on my local disk for which I would like to create multimodal embeddings using Vertex API.  I am finding that the largest MP4 file I can send is about 240 seconds. I thought that the documentation said if you specified a start and end time you could handle video files of arbitrary length. Am I doing something obviously wrong or do arbitrary length files only work if they are located on Google Cloud storage? 

[dawnberdan]

Hi @picheny,

Welcome to Google Cloud Community!

You're right! Vertex AI documentation confirms that specifying start and end times allows management of video files of any length, whether stored locally or in Google Cloud Storage. If you're encountering a 240-second limit with local files, potential factors include HTTP request size limits, client library constraints on file size, and network stability issues causing upload timeouts.

Here are some troubleshooting steps:

1. Stream the video in smaller segments using libraries like requests.post with stream=True in Python.
2. Upload your  videos to Google Cloud Storage for seamless integration with Vertex AI, efficient file handling, and reliable storage.
3. Ensure you're using the latest Vertex AI client library version to address any file size limitations.
4. Optimize network stability to prevent interruptions during uploads.

For understanding Vertex AI's video processing capabilities, you can explore these resources:

• Core Vertex AI Overview: Introduction to the platform and its features.
• Video Classification with AutoML Tutorial: Guides on project setup, dataset creation, model training, and batch predictions. 
• Get Video Descriptions: Focuses on using Vertex AI for video description tasks. Covers obtaining predictions, including shot-level and one-second interval classification.

I hope the above information is helpful.

[picheny]

I am using the "get_embeddings" api inside of the vertex API. You pass it a "video" object. If the object is on google storage everything works, but if the file is remote, it fails if the file is too large. I don't see how to apply solution 1, file transmission is buried somewhere in the API. 

[dawnberdan]

Hi @picheny,

You're right that the "get_embeddings" API in Vertex AI doesn't support streaming video data from local files. This makes it difficult to handle large videos stored locally because the API requires all video data to be provided at once, without the ability to handle smaller segments or streams. This limitation is likely due to the API's internal processing of video data as a single entity, which would need significant changes to support streaming video segments effectively.

Using Cloud Storage is ideal,  Uploading your videos to Google Cloud Storage is highly recommended. It offers scalability, reliability, and seamless integration with Vertex AI's "get_embeddings" API.

Here are some approaches to consider:

• Review the documentation of your client library (e.g., Python) for any file size limits when using the "get_embeddings" function with local videos. Consider switching to alternative libraries if there are constraints, offering better support for larger uploads or finer control over file segmentation during upload.
• If you encounter limitations with the client library, consider splitting the large video file into smaller segments locally using tools like OpenCV or ffmpeg. Upload each segmented chunk individually to Vertex AI using the "get_embeddings" function to bypass any file size restrictions imposed by the client library
• Temporarily upload the large video file to Google Cloud Storage. Utilize the "get_embeddings" function with the video's URL in Cloud Storage for processing. Once processing is complete, remove the temporary Cloud Storage file.

I hope the above information is helpful.