text-embedding-004 not working but textembedding-gecko working fine. Throwing the following error

[krsudhir]

In batch prediction job. The following error is thrown

INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/1039790610204/locations/us-central1/batchPredictionJobs/3180560206245396480 current state: JobState.JOB_STATE_QUEUED INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/1039790610204/locations/us-central1/batchPredictionJobs/3180560206245396480 current state: JobState.JOB_STATE_QUEUED INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/1039790610204/locations/us-central1/batchPredictionJobs/3180560206245396480 current state: JobState.JOB_STATE_FAILED.

Error log:

    249         # JOB_STATE_FAILED or JOB_STATE_CANCELLED.
    250         if self._gca_resource.state in _JOB_ERROR_STATES:
--> 251             raise RuntimeError("Job failed with:\n%s" % self._gca_resource.error)
    252         else:
    253             _LOGGER.log_action_completed_against_resource("run", "completed", self)

RuntimeError: Job failed with:
code: 13
message: "INTERNAL"

I did the changes for bucket permissions as suggested. There is no difference.  If I point to different model like the earlier "gecko" it works fine. like below.

textembedding_model = TextEmbeddingModel.from_pretrained("textembedding-gecko")

#textembedding_model = TextEmbeddingModel.from_pretrained("text-embedding-004"). You

can verify it in the output folder.

Display Name : BatchPredictionJob 2024-07-27 09:58:12.649290
Resource_name : projects/1039790610204/locations/us-central1/batchPredictionJobs/3741582715783217152

INFO:google.cloud.aiplatform.jobs:Creating BatchPredictionJob

Job state : JobState.JOB_STATE_SUCCEEDED

[jeerahalli]

I think I have found the issue, could you please grant the service account a role with the next two permissions to the bucket 

• storage.objects.get 
  storage.objects.create

The role `Storage Object Admin` might work. After performing this action, please try to run the pipeline again. 

[krsudhir]

I did the changes for bucket permissions as suggested. There is no difference.  If I point to different model like the earlier "gecko" it works fine. like below.

textembedding_model = TextEmbeddingModel.from_pretrained("textembedding-gecko")

#textembedding_model = TextEmbeddingModel.from_pretrained("text-embedding-004"). You

can verify it in the output folder.

Display Name : BatchPredictionJob 2024-07-27 09:58:12.649290
Resource_name : projects/1039790610204/locations/us-central1/batchPredictionJobs/3741582715783217152

INFO:google.cloud.aiplatform.jobs:Creating BatchPredictionJob

Job state : JobState.JOB_STATE_SUCCEEDED

[jeerahalli]

Hello,

Thank you for contacting the Google Cloud Community.

I have gone through your reported issue, however it seems like this is an issue observed specifically at your end. It would need more specific debugging and analysis. To ensure a faster resolution and dedicated support for your issue, I kindly request you to file a support ticket by clicking here[1]. Our support team will prioritize your request and provide you with the assistance you need.

For individual support issues, it is best to utilize the support ticketing system. We appreciate your cooperation!

[1]: https://cloud.google.com/support/docs/manage-cases#creating_cases