text-embedding-004 not working but textembedding-gecko working fine. Throwing the following error

[krsudhir]

In batch prediction job. The following error is thrown

INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/1039790610204/locations/us-central1/batchPredictionJobs/3180560206245396480 current state: JobState.JOB_STATE_QUEUED INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/1039790610204/locations/us-central1/batchPredictionJobs/3180560206245396480 current state: JobState.JOB_STATE_QUEUED INFO:google.cloud.aiplatform.jobs:BatchPredictionJob projects/1039790610204/locations/us-central1/batchPredictionJobs/3180560206245396480 current state: JobState.JOB_STATE_FAILED.

Error log:

    249         # JOB_STATE_FAILED or JOB_STATE_CANCELLED.
    250         if self._gca_resource.state in _JOB_ERROR_STATES:
--> 251             raise RuntimeError("Job failed with:\n%s" % self._gca_resource.error)
    252         else:
    253             _LOGGER.log_action_completed_against_resource("run", "completed", self)

RuntimeError: Job failed with:
code: 13
message: "INTERNAL"

I did the changes for bucket permissions as suggested. There is no difference.  If I point to different model like the earlier "gecko" it works fine. like below.

textembedding_model = TextEmbeddingModel.from_pretrained("textembedding-gecko")

#textembedding_model = TextEmbeddingModel.from_pretrained("text-embedding-004"). You

can verify it in the output folder.

Display Name : BatchPredictionJob 2024-07-27 09:58:12.649290
Resource_name : projects/1039790610204/locations/us-central1/batchPredictionJobs/3741582715783217152

INFO:google.cloud.aiplatform.jobs:Creating BatchPredictionJob

Job state : JobState.JOB_STATE_SUCCEEDED