PDFs in Datastore for Agent Builder are not getting Indexed

[acapone]

I am using the Agent Builder and creating a datastore of PDFs, but I keep getting an error that says the PDFs are not getting indexed. Has anybody been able to get past this before?

Here is the full log of the error, it is the same for every PDF that I have in the datastore:

{
"insertId": "1ipuwtkd1yzi",
"jsonPayload": {
"serviceContext": {
"service": "discoveryengine.googleapis.com"
},
"context": {
"reportLocation": {
"functionName": "google.cloud.discoveryengine.v1alpha.DocumentService.ImportDocuments"
}
},
"status": {
"message": "Document projects/hidden/locations/us/collections/default_collection/dataStores/test-gen-app_hidden/branches/0/documents/f286e6fc5e5a21733dd0a414d3a955af (uri: gs://test_gen_app_mxa/3-bayesian-network-inference-algorithm.pdf) is imported but not yet indexed. Its index status is still pending.",
"code": 14,
"@type": "type.googleapis.com/google.rpc.Status"
},
"importPayload": {
"gcsPath": "gs://test_gen_app_mxa/3-bayesian-network-inference-algorithm.pdf",
"lineNumber": "0",
"operation": "projects/hidden/locations/us/collections/default_collection/dataStores/test-gen-app_hidden/branches/0/operations/import-documents-4937549450197619152",
"document": ""
},
"@type": "type.googleapis.com/google.cloud.discoveryengine.logging.ErrorLog"
},
"resource": {
"type": "consumed_api",
"labels": {
"project_id": "hidden",
"version": "",
"location": "us",
"service": "discoveryengine.googleapis.com",
"method": "google.cloud.discoveryengine.v1alpha.DocumentService.ImportDocuments",
"credential_id": ""
}
},
"timestamp": "2024-10-31T18:33:24.732400Z",
"severity": "ERROR",
"logName": "projects/hidden/logs/discoveryengine.googleapis.com%2Fapi_errors",
"receiveTimestamp": "2024-10-31T19:18:15.833065327Z"
}

[ombatt]

I'm also experiencing this kind of error: 

"Document projects/359174615296/locations/global/collections/default_collection/dataStores/datastore-vdt_1730805380076/branches/0/documents/bc9aeb370ef4f5698eaf945bcdcf8c02 (uri: gs://bucket_vdt/VDT.txt) is imported but not yet indexed. Its index status is not found."

I had delete and create the bucket and the datastore many times without success. The txt file is very simple, just few word to make some tests.

[badcom]

I've been getting the same error with an xlsx file. The agent can find responses in the data store even though that error is displayed.

[acapone]

Yeah, I have the same experience. There are errors on all of my files, yet the agent does search through them and find responses. 

I am curious if the error is real or some kind of bug? Would love to have an answer before I try productionalizing anything with this tool.

[MarvinLlamas]

Hi @acapone,

Welcome to Google Cloud Community!

It appears that you are encountering an error because the PDFs within the datastore are not being indexed when using Vertex AI Agent Builder. To resolve this issue, you may use the getDocument method to verify again the indexing status of your documents.

If you continue to run into issues, consider reaching out to Google Cloud Support to further check underlying issues. When you contact them, be sure to provide as much detail as possible and include screenshots. This will help them understand your problem better and get it sorted out more quickly.

I hope the above information is helpful.