Internal error occurred while fine-tuning gemini flash 2.0 model.

[naveenselvam17]

I have started the fine-tuning for the gemini flash 2.0 model. After running it for around 8 hours, I got the error "Internal error occurred. Contact Vertex AI.". Attaching the error image and tuning details. 

[ruthseki]

Hi @naveenselvam17,

Welcome to Google Cloud Community!

While "Internal error occurred" is a general error and points to a problem on the Google Vertex AI side, here are a few things you can try:

• You may retry the fine-tuning job. Sometimes, these transient errors resolve themselves. Make sure to restart the job from the beginning.
• In terms of resource usage, Gemini 2.0 Flash use Dynamic shared quota (DSQ) which removes the need to set quotas and to submit quota increase requests. There are no quotas with DSQ. To help ensure high availability for your application and to get predictable service levels for your production workloads, see Provisioned Throughput.
• Consider reviewing your training data.** **While less likely to cause a generic "Internal error," ensure your training data is well-formatted and does not contain corrupted or malformed data points. Check for inconsistencies in data types, missing values (if not handled explicitly), or unexpected characters. Check if the training dataset is exceptionally large. Large datasets, especially if unoptimized, can sometimes trigger unexpected errors during processing.
• Verify fine-tuning configuration. Double-check all the configuration settings for your fine-tuning job in Vertex AI:
  • Number of Epochs: Ensure this is a reasonable value for your dataset and model.
  • Learning Rate Multiplier: Verify this is within an acceptable range. An extremely high or low learning rate can lead to instability and potential errors.
  • Adapter Size: Make sure the specified adapter size is compatible with the model and your resources.
  • Truncated Example Count: Confirm this is set appropriately for your use case.
• Simplify the training process (if possible):
  • Reduce Epochs: Try a smaller number of epochs (e.g., 1-2) to see if the problem occurs sooner. This can help isolate if the issue is related to the length of the training run.
  • Smaller Dataset (Subset): If feasible, train on a smaller subset of your data to see if the problem disappears. This would suggest an issue within your dataset.
  • Standard Configuration: If you've customized the training process significantly, revert to a more standard or default configuration to see if the problem persists.
• Look for logs related to your fine-tuning job or any Vertex AI services involved. The logs might contain more detailed error messages, stack traces, or other information that can provide clues.
• Check if there have been recent updates to Vertex AI or the Gemini Flash 2.0 model. Sometimes, bugs are introduced in new releases. If you suspect this, you might temporarily try using an older version of the Vertex AI SDK (if applicable) or a different model version (if available).
• Sometimes, problems can arise from corrupted environments or outdated dependencies. Try creating a fresh Vertex AI Notebook instance or setting up a new environment to run your fine-tuning job.

If none of these steps resolve the issue, contact Google Cloud Support. Provide as much detail as possible about the circumstances leading up to the error, including:

• Model you are fine-tuning (Gemini Flash 2.0).
• Configuration settings (like number of epochs, learning rate, adapter size, and truncated example count).
• How you initiated the fine-tuning process.
• Any custom code or modifications you made.
• The time it ran before the error occurred.
• If you have any specific error codes or logs associated with the "Internal error," include those in your report.
• Describe how this error is affecting your project.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.