Hitting quota limit when importing documents. No active LROs

[adriaan]

My ImportDocumentsRequests are being denied due to Quota limit 'ConcurrentImportDocumentsRequestsPerProjectEU' exceeded. The error description mentions that I already have 3 concurrent import operations running.

I was doing a bunch of trial-and-error work before the error arose. However, in contrast to what the error message claims, when I list the number of operations associated with the project, none have status 'running'. They are all either 'failed' or 'canceled'. I don't see how I can be hitting the limit then.

Am I missing something or is this an error on gcloud's side?

[adriaan]

I've opened an issue for this in the meantime https://issuetracker.google.com/issues/376173729.

Repeating my last comment on the issue, in case anyone else has also encountered this:

>> when the gcs_uri parameter of GcsDocument does not start with 'gs://', the request fails and the quota usage increments by one and stays there.

[ibaui]

Hi @adriaan,

Welcome to Google Cloud Community!

Based on your description, it seems you're encountering a discrepancy between the error message and the actual status of your import operations. It's possible that there might be a delay in updating the status of these operations. Sometimes, operations may still be counted toward the quota limit even if their status is not immediately reflected as 'running.' Here are some potential reasons and suggestions you might consider to address the issue: 

Delayed Status Updates: DocumentAI operations, especially imports, can take time to complete. The error message might be reflecting a temporary state where operations were running concurrently, even though they've since transitioned to "failed" or "canceled." The status updates might be lagging behind the actual operation state. Give it some time for the status of your previous operations to update, and consider submitting your ImportDocumentsRequest again after a while.

Quota Limits and Regional Considerations: 

• Regional Quotas: The "ConcurrentImportDocumentsRequestsPerProjectEU" quota specifically applies to import operations within the EU region. If you're performing imports in other regions, they might not be affected by this particular quota.
• Check Quota Limits: Since you mentioned that you are doing a lot of trial-and-error work, it means you might be starting and stopping operations frequently, which can quickly use up your allotted concurrent requests. Even if the jobs were canceled or failed, they might still count against your quota for a period. With this, review your project's quota limits for import operations to ensure that you're not consistently exceeding them.You can confirm whether the quota limit in your Google Cloud project has indeed been exceeded. You can navigate to the Google Cloud Console, and, in the left-hand navigation panel, click on "IAM & Admin" and then select  “Quotas & System Limits."  You can filter by the specific service that might be exceeded. If necessary, you may need to request a quota increase. Keep in mind that this request is subject to review and approval, and it may take some time to process.

You can also visit the following documentation for more details:

• Document AI Quotas
• Document AI System Limits
• Document AI Overview

I hope the above information is helpful.

[adriaan]

Hi there, it's been almost a week and I'm afraid the quota usage is still stuck.

The behavior of the API is very surprising: a malformed request doesn't return a 422 but a 502. Next,  despite triggering an internal error, the request gives rise to a new operation, or at least it seems to, if you ask our quota usage. Strangely, the operation doesn't appear anywhere I can look, so even if it exists I can't cancel it or check its status. After more than 48 hours, the reported quota usage remains unchanged, which gives the impression that somewhere, somehow, the operation a) still exists or b) its non-existence hasn't been registered in the database.

We've increased the quota limit, but so long as this behavior persists, I'm reluctant to send any more requests to the API, for fear of locking myself out of importing documents again.