Internal Error Occurred; Contact Vertex AI

[jsniff12]

Hi,

I've been doing some fine-tuning runs and I keep getting a strange error where there's little information on what the actual issue was. It simply says "Internal Error Occurred; Contact Vertex AI"; pls see attached snapshot. Any feedback on what the actual issue is? I don't see anything useful in the logs and oddly enough, the error sometimes happens and sometimes does not without changing anything on my end (which leads me to believe there's something wrong with the service and not my model file).  Anyone see this issue? I don't think it's any formatting or capacity issue; I've checked that. 

Also odd is that the error often occurs 75%+ the way through a multi-hour run.

Best Regards,

Jacob

[cassandramae]

Hi @jsniff12,

Internal server errors can sometimes occur due to temporary issues on the server side or with the service itself. With this, it's advisable to report this issue to Google Cloud Support. They can provide more insights into whether this is a known issue, whether it's specific to your project, or if it's a widespread problem affecting other users as well.

You can also file an issue tracker using this link.

[dainal12]

This error could be due to intermittent service disruptions on Vertex AI's end, especially if it consistently occurs late in the run; consider reaching out to Google Cloud support with the job ID for further investigation.