Custom Training Job at Vertex AI fails

[sureshAZ]

Dear all,

I am trying to fine tune a model in Vertex AI using custom training option. During the execution, I had observed 3 issues.

1. The model is not stored in GCS despite giving the following options.

• output_dir specification in TrainingArguments of Transformers library
• train.save_model(output_dir)

2. During the training when the model reports its progress as shown in the screenshot, it flags that line as Error though there is no error message shown. What's the error here? How do I understand this msg or resolve this? There are 100s of failures similar to this which don't appear to be truly a failure.

3. The job shows failed at the end without specifying the reason. Is it because of all these errors reported as shown in 2.

Please advise. Thanks for your help.

Suresh

[MarvinLlamas]

Hi @sureshAZ,

Welcome to Google Cloud Community!

It seems that you are encountering an issue in your Vertex AI custom training job that fails silently, displays incorrect “ERROR” messages in the logs, and doesn’t save the model to Google Cloud Storage, probably due to a bug in your training script (run_module.py) that causes a crash before the model is saved.

Here are some potential ways to address your issue:

• Simplified Training: Begin by training a small model on a small dataset to identify any core issues. If it works, gradually scale up the complexity of your model and dataset. 
• Examine complete logs: You may want to download and review the full job logs from Vertex AI, as they may contain more information than what you see in the console. Be sure to check for any error messages or warnings, even if they seem minor.
• Check for resource limits: You might want to review the job’s metrics in the Vertex AI console to identify any patterns or anomalies. Look for inconsistencies that may indicate the root cause of the issue.

You can refer to the following documentation to understand the components of your setup and gather the necessary information for potential solutions to the issue.

• Custom training overview
• Create custom jobs
• Cloud storage permission

I hope the above information is helpful. 

[chunlexiong]

I have exactly the same issue: every progress is shown as an error, but Vertex AI never points out what the error is. And my training can finish successfully and save the trained model successfully. How can I download the full job logs from VertexAI? @MarvinLlamas