Instructions in vertex ai agent

[Calvinstp]

I'm building a Sales Assistant for the company I work for. I'm using Vertex AI Agent and using Playbooks & Tools.

How do you write instructions and objectives correctly? Can I create an agent that only retrieves answers from the Tools Data Store and does not generate other answers that are not in the data store?

Because I experience problems, often the answers given do not match the knowledge base I provide. and when I checked the conversation history, the answer given was not the result of checking the tools.

I have followed the tutorial from the following link https://www.youtube.com/watch?v=cQfl3Nl3RXA and still have not found a solution to the problem I am facing.

[ibaui · Accepted solution]

Hi @Calvinstp,

Welcome to Google Cloud Community!

You're taking the right approach by using Vertex AI Agent and Playbook Tools to build your Sales Assistant. It's common to encounter issues where the agent generates answers that don't match your knowledge base, especially when you want it to rely solely on the Data Store Tools. In light of this, consider the following to resolve the issue:

Writing Effective Instructions and Objectives:

• Focus on User Intent: Craft instructions that capture the user's intent behind their questions. Think of specific scenarios and questions they might ask.
• Actionable Objectives: Define objectives that guide the agent towards retrieving specific data from the Data Store Tools. These objectives should be measurable and relevant to the instructions.
• Examples:
•    Instruction: "Provide information about product pricing when the user asks about 'pricing' or 'cost'."
•    Objective: "Retrieve data from the Data Store Tools containing 'price' information when the user's query contains keywords like 'pricing' or 'cost'

Using Playbook Tools Effectively:

• Data Store Tools: Ensure that your Data Store Tools is properly configured and contains the relevant information for your Sales Assistant.
• Playbook Structure: Structure your playbook to guide the agent through the steps of retrieving information from the Data Store Tools.

Controlling Agent Behavior:

• Prompt Engineering: Craft your prompts carefully to guide the agent towards retrieving information from the Data Store Tools.
• Temperature and Top-P: Experiment with different Temperature and Top-P values to control the agent's creativity and tendency to generate responses that are not based on the Data Store Tools. Lower values generally lead to more factual and data-driven responses.

Enforcing Knowledge Base Adherence:

• Confidence Threshold: Set a high confidence threshold for retrieving answers from the Data Store Tools. This minimizes the risk of the agent generating responses based on general knowledge when the confidence in its match with the data store is low.
• Fallback Handling: Define a fallback action when the agent's confidence in retrieving a response from the data store is below the threshold. This can involve prompting the user to rephrase their question, offering a limited selection of pre-defined responses, or transferring the conversation to a live agent.

Just to add, Playbook tools are currently in Preview, which means it is still under development and may not yet offer the expected quality. Preview features often come with certain limitations and might not provide the full range of quality or functionality available in the final product. It may have bugs or unexpected behaviors. You can expect the quality to improve as the feature matures. 

I hope the above information is helpful.

[Calvinstp]

I finally switched to using Agent type Chat and using Dialogflow CX. I am aware that the conversational agent provided is not yet perfect, it is still in the development stage.

But thank you for the answers and suggestions that have been given. will be very helpful when we migrate back to conversational agents.