Agent Builder Search App unable to generate lists

[djimenez]

Hi all,

I have deployed an Agent Builder App Search with 2.5K Jsonl metadata + unstructured PDF documents. Before importing to discovery engine, I have extracted metadata such as "client", "services", etc... using methods like regex, gemini querying, etc... for each document. I can see that metada have been correctly assigned to each object and is retrievable, indexable, dynamic facetable and searchable. I have added this fields as "dynamic facet" and added the filters (in fact, the client facetable filter show all the clients correctly). All inidcates that discovery engine has imported correctly the metadata. But, if I try to ask an easy question like: "List all the clients in the documentation"  my agent is unable to do it. I have tried modifying the prompt in every possible way (following suggestions of Gemini, chatgpt, etc...), but no one did the trick. Anyone have faced the same problem? Thanks!

[ruthseki]

Hi @djimenez,

Welcome to Google Cloud Community!

The problem likely lies not in the data import or indexing into the Discovery Engine, but rather in how your Vertex AI Agent Builder is interacting with the Discovery Engine and formulating its response. Even though your metadata is correctly indexed and facetable, the agent's prompt and its understanding of how to use the Discovery Engine's capabilities might be insufficient.

Here are some workarounds that you may try:

1. Prompt Engineering: 

• Your prompt "List all the clients in the documentation" is too general. The agent might not understand the context of "documentation" in relation to your Discovery Engine. Try these more specific prompts:
• "List all unique values for the 'client' field from the Discovery Engine search results."
• "Retrieve a list of all distinct clients mentioned in the documents indexed in the Discovery Engine."
• "Use the Discovery Engine to provide a list of all clients, based on the 'client' dynamic facet." (This explicitly mentions the facet.)
• Explicitly instruct the agent to use the Discovery Engine: Clearly state that it should leverage the Discovery Engine for this task. Example: "Using the Discovery Engine's capabilities, list all unique clients found in the indexed documents."
• Specify the desired output format: Tell the agent how you want the list formatted. Example: "Return a comma-separated list of all unique clients found in the indexed documents." or "Return a bulleted list of all unique clients found in the indexed documents."

2. Agent Configuration and Knowledge Base:

• Verify Agent Access: Double-check that your Agent Builder has the necessary permissions to access and query your Discovery Engine instance.
• Knowledge Base Integration: Make sure your Agent Builder's knowledge base is correctly configured to use your Discovery Engine as a data source. There might be specific configuration parameters or API keys required to establish this connection.
• Contextual Understanding: The agent may lack the understanding of how to translate a natural language question into a specific Discovery Engine query. You might need to provide more context in the prompt or in a dedicated knowledge base section explaining how the Discovery Engine is structured and how to query it.

3. Discovery Engine Querying:

• Query Limits: If you have a large number of clients, you might need to paginate the results.
• Field Name: Confirm that "client" is the exact name of your field in the Discovery Engine. Case sensitivity matters.

4. Debugging Steps:

• Simplify: Try a very simple test case. Create a small, sample dataset with a few known clients and test your agent against that to isolate whether the issue is with the data or the agent's interaction with the data source.
• Logging and Monitoring: Enable logging in your Agent Builder to understand the queries it's sending to the Discovery Engine and the responses it's receiving. This will pinpoint where the process breaks down.
• Direct Discovery Engine Query: Bypass the agent and directly query the Discovery Engine using its API. If you can successfully retrieve the list of clients via the API, then the problem lies solely with the agent's prompt or configuration.

Here are some useful links to the Vertex AI Agent Builder documentation that might help you provide detailed guidance on setting up and configuring your search app, including how to handle metadata and queries:

• Introduction to Vertex AI Search
• What is Vertex AI Agent Builder?
• Get started with generic search
• APIs and Reference for Vertex AI Agent Builder

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[djimenez]

Hi Ruthseki,

First of all, many thanks for taking the time to answer my question.

Regarding the proposed actions:

Prompt Engineering:

• I have tried the proposed prompts and added more variations. I also tested translating them into Spanish (the current default language) and changing the default language to English while asking the prompts in English. The usual response to Spanish prompts is: "The answer cannot be generated." In English, the result is more or less the same.

• Clearly stating that Discovery Engine capabilities or dynamic facetable fields should be used (ensuring that we use the exact terms present in the facetable and metadata fields) did not help either.

Agent Configuration:

• The agent was created using the step-by-step method in the Agent Builder console (I only rely on the API to automate imports). From my point of view, if the agent is able to retrieve exact information from the data source, then the connection and permissions should be correct.

• Regarding contextual understanding, I have tested several prompts describing how the metadata is structured inside the JSONL file. For instance:

  "Each indexed document contains a set of metadata. One of the metadata fields is the 'client' field. This field is retrievable and searchable. I need you to use Discovery Engine to create a list of all values for the 'client' metadata field across all documents."

  Result: "A summary could not be generated for your search. Here are some of the search results."

Debugging Steps:

• Simplify: From my point of view, 150 clients and 2,400 documents should be a small enough dataset, but we will create a smaller dataset to verify this.

• Logging and Monitoring: The documentation you referenced is for Dialogflow (in fact, I have also been testing Conversational Agents, and debugging playbooks is great with this feature). However, I have not been able to find a similar setting in the "Agent Builder" App Search. I managed to enable audit logs for Agent Builder, but they only log the API request itself, without including the payload (i.e., the answers, responses, or reasoning behind them are not recorded in the logs). Perhaps there is an API call to enable this feature in App Search / Datastore, but we haven't found it yet.

• Bypassing the agent and querying the API directly: The Discovery Engine is unable to retrieve the client list, and the output is just a list of pages. However, the returned pages do contain metadata, which suggests that the metadata is being indexed but is not being used in the search results or response generation.

From my point of view, the main issue seems to be that Agent Builder relies solely on unstructured data (PDF documents) to search and generate responses, without taking metadata into account for result retrieval or answer generation.

Best regards,
David