How to make sure the agent answers only based on the data from the tool

[AnanthM]

I'm developing a Vertex AI agent using the agent builder to answer user questions about investment products. The agent covers 12 products across equity, fixed income, real estate, private equity, and real assets. It uses an OpenAPI-based tool to access the dataset and respond to queries.

The agent sometimes fabricates answers when asked questions that are outside its dataset, instead of admitting it doesn't know. This is particularly problematic as it's dealing with financial information.
Here are questions it can answer well. All these are in our dataset and hence it can answer well. 

• Q: What equity products do you have?
• Q: What is the 5yr return for the global equity fund?
• Q: What products do you have that invest in real estate?
  
  Here are some problematic examples. 
• Q: What is the 5yr return for your small cap fund? (It provides a return, but we don't have a small cap fund)
• Q: What is the fund with the maximum return? (It doesn't return the correct fund, just selects one arbitrarily)
  
  I tried the following to solve the problem. 
  I updated the goal to say you should only answer based on the tool. I also updated instructions to say "if the tool doesn't have the answer, just say you don't know.
  I provided examples teaching it how to answer these questions. It was able to learn for those examples but there were other questions in a similar vein it still gets wrong. 

So here are my questions. 
1. How can we instruct the agent to rely solely on data in the dataset and not fabricate answers?
2. Is it possible to instruct the agent to refuse to answer questions about specific topics within the domain?
3. Are there best practices for preventing AI agents from making up answers in financial contexts?
4. Has anyone successfully implemented a "confidence threshold" for Vertex AI agents?
5. Are there specific prompting techniques that work well for maintaining strict boundaries on an agent's knowledge?

Any insights, experiences, or suggestions would be greatly appreciated. Thank you in advance for your help!

[Mo5ito]

i'am having the same issue. the bot is responding random stuff without limit

[dawnberdan · Accepted solution]

Hi @AnanthM,

Welcome to Google Cloud Community!

You're encountering a common challenge in AI development. One of the common challenges for an agent is by strictly following the knowledge base and refraining from inventing information. This is especially crucial in financial sectors where accuracy is extremely important and significant.

Here's a breakdown of your questions and potential solutions:

1. Relying Solely on Dataset Data

• Use clear and specific instructions that emphasize reliance on the tool's data:
  e.g. "Your responses must be based solely on information retrieved from the tool. If the tool doesn't have the answer, say 'I don't have information on that."
  
• Include negative examples in your training data. Show the agent what not to do.
  e.g.   Question: What is the 5 year return for your small cap fund?
            Answer: I don't have information on a small cap fund.
  
• Instruct the agent to only respond to queries about the 12 specific products. If a user asks about something outside that range, it should respond with "I can only provide information about the following products: [list of products]."

2. Refusing to Answer Specific Topics

• Instruct the agent to recognize certain topic keywords and decline to answer questions related to them. For example, If the question is about a 'small cap fund,' respond with “I'm not able to provide information about small cap funds."

3. Best Practices for Preventing Fabrication

• Design your tool to provide verifiable information whenever possible. This might mean including links to sources or allowing the agent to cite specific data points.
• Include clear disclaimers stating that the agent's knowledge is limited to the provided data.
• You can also allow users to flag responses that appear fabricated or misleading, which can help improve the agent's accuracy.
• You can also check this document for best practices.
  

4. Confidence Thresholds

• Some large language models (LLMs) offer confidence scores for their responses. You might be able to integrate this mechanism into your Vertex AI agent. If the confidence score falls below a certain threshold, the agent could decline to answer.
• Implement checks within your tool or agent logic to assess the validity of the information being returned. If the information seems inconsistent or lacks sufficient detail, the agent could decline to answer.
  

5.Techniques for Boundary Maintenance

• Use clear and structured prompts that emphasize the importance of data-driven responses. For example, "Using the information from the tool, answer the following question:..."

Additionally, ensure your dataset is comprehensive, accurate, and up-to-date and continuously monitor the agent's performance and adjust your training data, instructions, and prompts as needed. 

By implementing these strategies and consistently testing and refining your agent, you can improve its accuracy and reliability, minimizing the risk of fabricated answers in a financial context.

I hope the above information is helpful!