Document AI, Custom Classifier - Internal Error when Training

[BruceInfodev]

I have created a custom classifier in document AI.

I have reduced the labels, training documents, and testing documents to the minimum allowed to get the classifier to train, but it always fails.

• Labels = 2
• Training Documents = 10
• Testing Documents = 2

Training fails with the following error:

{
  "name": "xxxxxxxxxxxxx",
  "done": true,
  "result": "error",
  "response": {},
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.documentai.uiv1beta3.TrainProcessorVersionMetadata",
    "commonMetadata": {
      "state": "FAILED",
      "createTime": "2024-11-08T15:23:31.941368Z",
      "updateTime": "2024-11-08T15:33:52.389110Z",
      "resource": "xxxxxxxxxxxxxxxxx"
    },
    "trainingDatasetValidation": {},
    "testDatasetValidation": {}
  },
  "error": {
    "code": 13,
    "message": "Internal error encountered.",
    "details": []
  }
}

and in Google Cloud Console Logs Explorer:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 3,
      "message": "The replica workerpool0-0 exited with a non-zero status of 1. Termination reason: Error. To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=xxxxxxx&resource=ml_job%2Fjob_id%2Fxxxxxxxxxx&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%xxxxxxxxx%22"
    },
    "authenticationInfo": {
      "principalEmail": "xxxx@xxxx.com"
    },
    "requestMetadata": {
      "requestAttributes": {},
      "destinationAttributes": {}
    },
    "serviceName": "documentai.googleapis.com",
    "methodName": "google.cloud.documentai.uiv1beta3.DocumentProcessorService.TrainProcessorVersion",
    "resourceName": "xxxxxxxxxxxxx"
  },
  "insertId": "1y8j6lyc8cm",
  "resource": {
    "type": "audited_resource",
    "labels": {
      "method": "google.cloud.documentai.uiv1beta3.DocumentProcessorService.TrainProcessorVersion",
      "project_id": "xxxxxxxxxxx",
      "service": "documentai.googleapis.com"
    }
  },
  "timestamp": "2024-11-08T15:11:02.914116110Z",
  "severity": "ERROR",
  "logName": "projects/xxxxxxxxxxx/logs/cloudaudit.googleapis.com%2Factivity",
  "operation": {
    "id": "projects/xxxxxxxxx/locations/xx/operations/xxxxxxxxxxxx",
    "producer": "documentai.googleapis.com",
    "last": true
  },
  "receiveTimestamp": "2024-11-08T15:11:03.812489435Z"
}

I do not know where to get any additional details, please advise.

[Aaryan]

Hello @BruceInfodev,

It seems that your training job is failing due to an internal error. I recommend reviewing your training data to ensure the labels are well-defined and that you have at least 20–30 documents, as 10 may be insufficient. Also, check your Google Cloud resources under IAM & Admin > Quotas to ensure you have enough CPU and memory, and consider retraining with a smaller subset of data to see if that resolves the issue.

[MarvinLlamas]

Hi @BruceInfodev,

Welcome to Google Cloud Community!

It appears that you are encountering an error while training your documents using a custom classifier in Document AI. Possible contributing factors may include resource availability and incorrectly labeled or formatted data.

Here are some potential ways to address your issue:

• Check the Logs: You can view your audit logs in the Google Cloud Console's Logs Explorer and filter them by the service name 'documentai.googleapis.com' and the method 'TrainProcessorVersion' to find additional details about why the training job failed. This might provide more context regarding the failure.
• Assess the Data: Verify that your training and testing documents are properly labeled and formatted. This might help correct the inconsistencies in your data that cause the training to fail.
• Review resource limits: Check if you’re hitting any API rate limits or quota restrictions in your Google Cloud project, as they may lead to training job failures.
• Training setup options: Ensure that your processor is properly configured and initialized, as it may help ensure the successful training of documents.

if you continue to run into issues, consider reaching out to Google Cloud Support to further check underlying issues. When you contact them, be sure to provide as much detail as possible and include screenshots. This will help them understand your problem better and get it sorted out more quickly.

I hope the above information is helpful.

[BruceInfodev]

I've checked all the methods mentioned above, but the error still occurred after retraining. However, after just a few hours, I retried the training process, and it was successfully completed. Although I still don't know the exact reason for this.