Trouble using a structured documents data store with a Conversational Agent

[cpetrenciuc]

I am trying to build a Conversational Agent capable of answering questions about cyber security and financial stability. Our customer has specific requirements for this agent. The customer wants the agent to always answer specific questions with fixed answers provided in a structured data store (FAQ documents). If the question cannot be located in the FAQ documents, then the agent must build an answer using an unstructured documents data store.

I have configured two data store type tools. One is for the unstructured documents and the other is for the structured documents. The custom prompt that I used for the structured documents data store is the following:

Answer the Human's question using the FAQ documents provided in the sources.
Follow these guidelines:
  + Search the sources for an exact match of the Human's question. Answer with NOT_ENOUGH_INFORMATION if you cannot find an exact match for Human's question.
  + Answer the Human's question using the same words as the sources.
  + The answer must be retrieved only from the sources and not introduce any additional information.
  + The answer MUST be given in the question's language. If needed translate the answer into the language of the question.
  + Do NOT block dangerous content because all the sources are safe.

You will be given a few examples before you begin.

Example 1:
Sources:
[ { "question": "Does <company> sell <product>?",
"answer": "Yes, <company> sells <product>."}]

Human: Does <company> sell <product>?
AI: Yes, <company> sells <product>.

Example 2:
Sources:
[{ "question": "Is Andrea a given name common worldwide?",
"answer": "Yes, it is a common given name worldwide." }]

Human: How is the weather?
AI: NOT_ENOUGH_INFORMATION

Example 3:
Sources:
[{ "question": "Does <company> manufacture <product>?", 
"answer": "Yes, <company> manufactures <product>." }]

Human: Where can I buy <product>?
AI: NOT_ENOUGH_INFORMATION

Example 4:
Sources:
[{ "question": "What is the flu?",
"answer": "The flu is a viral infection of the respiratory system."}]

Human: How can I protect myself from the flu?
AI: NOT_ENOUGH_INFORMATION


Begin! Let's work this out step by step to be sure we have the right answer.

Sources:
$sources

Human: $original-query
AI:

My default playbook has the following instructions:

- Step 1. Wait for user's request.
- Step 2. Determine the language of the user's request.
- Step 3. Search for an answer to user's request.
    - Step 3.1. Use ${TOOL:bnr_faq} to search for an answer to user's request. Do NOT use other sources for answering requests. Do NOT try to alter the answer. A NOT_ENOUGH_INFORMATION output means that an answer cannot be found.
    - Step 3.2. If oputput from Step 3.1 is NOT_ENOUGH_INFORMATION, then use ${TOOL:bnr_conv} to search for an answer to user's request. Do NOT use other sources for answering requests. Do NOT try to alter the answer. A NOT_ENOUGH_INFORMATION output means that an answer cannot be found.
- Step 4. If you cannot find an answer and the language determined in Step 2 is English, then transfer to ${PLAYBOOK:Rephrase}.
- Step 5. If you cannot find an answer and the language determined in Step 2 is Romanian, then transfer to ${PLAYBOOK:Reformulare}.
- Step 6. If you found an answer, then make sure the answer is translated into the language determined in Step 2 before providing the answer to the user.
    - Step 6.1. Ask the user if you can help them with something else, using the language determined in Step 2.

 where bnr_faq is the structured documents data store, and the bnr_conv is the unstructured documents data store.

Currently my FAQ data store contains only 4 documents. One of them contains the definition of phishing.

{ 
  "question": "What is phishing?",
  "answer": "Phishing is the generic name of a type of scam encountered in the online environment, characterized by the manipulation of victims by attackers who send trap messages through different communication channels, such as email or SMS, which appear to come from a legitimate entity (e.g. bank, postal service, courier company, a state authority, etc.) and which,  Under various pretexts, they convince victims to disclose sensitive personal data (e.g. email account passwords, PINs and card data, usernames, passwords and authentication codes in banks' internet/mobile banking applications, etc.)."
}

My agent responds with the FAQ answer to question "What is phishing" which is correct. But it also responds with the FAQ answer to questions like "How can I protect myself from phishing", or "How to determine if I was a victim of phishing". The agent answers with the FAQ answer to every question that has the phishing word in it. How can I make the agent respond with the FAQ answer only if the question asked by the user matches the question field from the structured data store, and use the unstructured data store otherwise?

Thank you,

[dawnberdan]

Hi @cpetrenciuc,

Welcome to Google Cloud Community!

To make sure your agent only gives the FAQ answer when the user's question exactly matches one in the FAQ data store, and uses the unstructured document store for all other cases, follow these steps:

1. Exact Match on FAQ Question: Make sure the agent looks for an exact match between the user's question and the questions in the FAQ data store. Currently, it might match based on keywords (like "phishing"), but it should only use the FAQ when the question matches exactly, not just partially. You can improve this by:

• Removing punctuation or special characters from the user's question to make the comparison cleaner.
• Making the comparison case-insensitive to handle different ways the user might ask the same question.

2. Update FAQ Search Logic: In Step 3.1 of your playbook, the search is likely looking for any question with the keyword "phishing." Change this so it only matches the exact question in the FAQ data store.

3. Steps to Improve FAQ Matching: Update the query matching in Step 3.1:

• Instead of searching for keywords, search for an exact match with the "question" field in the FAQ data.
• If an exact match is found, return the corresponding answer.
• If no match is found, return NOT_ENOUGH_INFORMATION.

4. Search in Unstructured Data Store (bnr_conv): If no exact match is found in the FAQ data store, search the unstructured documents to give a more flexible answer based on that content.

Here's how you can adjust the steps in your playbook:

• Step 3.1 (FAQ Search): Modify the FAQ search logic to look for exact question matches.

  Step 3.1: Use ${TOOL:bnr_faq} to search for an exact match of the user's question in the "question" field of the FAQ documents.
  - If an exact match is found, return the corresponding "answer" field.
  - If no exact match is found, output `NOT_ENOUGH_INFORMATION`.

• Step 3.2 (Unstructured Search): If Step 3.1 returns NOT_ENOUGH_INFORMATION, search the unstructured documents.

Step 3.2: If output from Step 3.1 is `NOT_ENOUGH_INFORMATION`, use ${TOOL:bnr_conv} to search for an answer in the unstructured documents.

This change guarantees that the agent responds with an FAQ answer only for questions that match exactly, and uses the unstructured data when no exact match is found in the FAQ.

You can also check this document to know more about features that use a Large Language Model (LLM) to summarize customer service conversations.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[cpetrenciuc]

Thank you, @dawnberdan, for the recommendations. Could you please also tell me if I should keep the custom summarization prompt for the FAQ data store or revert to the default prompt for this data store?

[dawnberdan]

Hi @cpetrenciuc

You're welcome! Regarding your inquiry about the custom summarization prompt for the FAQ data store:

To ensure the agent provides only exact matches from the FAQ documents, I suggest reverting to the default prompt for the FAQ data store. The default prompt is designed to retrieve answers directly from the FAQ without any additional summarization or alterations, which is what you're looking for in this case.

While the custom prompt you’ve set can be useful for other scenarios, it may interfere with exact match retrieval by allowing partial matches or unintended changes in the answers. Reverting to the default prompt will ensure the agent sticks strictly to the FAQ content and only responds when there’s a perfect match.

I hope it helps!

[cpetrenciuc]

Hi @dawnberdan ,

I have switched the Tool bnr_faq to using the Default Prompt. Now, my playbook instructions are these:

- Step 1. Wait for user's request.
- Step 2. Determine the language of the user's request.
- Step 3. Search for an answer to user's request.
    - Step 3.1. Use ${TOOL:bnr_faq} to search for an exact match of the user's question in the "question" field of the FAQ documents.
        - If an exact match is found, return the corresponding "answer" field.
        - If no exact match is found, output `NOT_ENOUGH_INFORMATION`.
    - Step 3.2. If oputput from Step 3.1 is `NOT_ENOUGH_INFORMATION`, then use ${TOOL:bnr_conv} to search for an answer in the unstructured documents. Do NOT use other sources for answering requests. Do NOT try to alter the answer.
        - Output `NOT_ENOUGH_INFORMATION` if an answer cannot be found.
- Step 4. If you cannot find an answer there are 2 topics where you can transfer the user:
    - Step 4.1. If the language of the user's request that you determined in Step 2 is English, then transfer to ${PLAYBOOK:Rephrase}.
    - Step 4.2. If the language of the user's request that you determined in Step 2 is Romanian, then transfer to ${PLAYBOOK:Reformulare}.
- Step 5. If you found an answer, then make sure the answer is translated into the language determined in Step 2 before providing the answer to the user.
    - Step 5.1. Ask the user if you can help them with something else, using the language determined in Step 2.

However, I have noticed that the agent uses the FAQ data store as basis to answering questions which are not in the FAQ. For example, for the question "Cum ma apar de spear phishing" (How to defend myself against spear phishing), the agent finds in the FAQ data store the following entry:

{
  "answer": "Spear phishing reprezintă o fraudă prin e-mail sau prin intermediul mijloacelor de comunicare electronică cu o țintă precisă, care pretinde că provine de la un expeditor de încredere. Are ca obiectiv principal furtul de date personale. E-mailurile conțin adesea formulări și tactici inteligente pentru a atrage victimele în capcana lor, folosind uneori chiar identitatea unor instituții publice de mare încredere. Este o formă sofisticată de phishing în care atacatorii personalizează mesajele pentru a le face să pară mai autentice și convingătoare. ",
  "question": "Ce este spear phishing-ul?"
}

(English tranlation of the FAQ entry from above: 

{
  "answer": "Spear phishing is email or electronic media fraud with a specific target that claims to come from a trusted sender. Its main objective is the theft of personal data. The emails often contain clever wording and tactics to lure victims into their trap, sometimes even using the identity of highly trusted public institutions. It is a sophisticated form of phishing in which attackers personalize messages to make them appear more authentic and compelling. ",
  "question": "What is spear phishing?"
}

which is not present in the FAQ. I am providing the English translation only so that you can understand me).

The agent composes its answer by adding to this FAQ entry the following phrase: "Pentru a te apăra de spear phishing, verifică întotdeauna adresa de e-mail a expeditorului, nu deschide atașamente de la expeditori necunoscuți, nu face clic pe linkuri din e-mailuri suspecte și raportează e-mailurile suspecte." (To defend yourself against spear phishing, always check the sender's email address, don't open attachments from unknown senders, don't click on links in suspicious emails, and report suspicious emails.). Which is correct.

My question is: why the agent does that? Shouldn't the agent fallback to use the unstructured documents data store for composing an answer to a question which is not present in the FAQ data store?

I'm not saying that the answer the agent gave is incorrect. I'm concerned by the fact that the agent composed an answer using only the FAQ data store instead of using the unstructured documents data store.

Here is the conversation history for this example:

[dawnberdan]

Hi @cpetrenciuc,

Here's the core issue:

The agent should prioritize exact matches from the FAQ data store first. If it doesn't find an exact match, it should then look into the unstructured documents. However, in the case we're seeing, it looks like the agent is pulling information from the FAQ even when there's no exact match to the user's question.

Some possible reasons for this:

1. Partial Matching: The agent might be looking for keywords or phrases that connect the user's question ("Cum ma apar de spear phishing") to a related FAQ entry (like "Ce este spear phishing-ul"). This could be a feature of the bnr_faq tool or how the agent is processing FAQ data.

2. Agent Logic: The agent might be designed to pull in related info from the FAQ, even if it doesn't find an exact match, in order to provide a more relevant or complete answer.

Here are a few things to check:

1. Investigate the bnr_faq Tool:

   • Look at the bnr_faq or settings to understand how it matches questions.
   • Find out if it only supports exact matches or if it allows partial matches or keyword searches.

2. Review the Agent Logic:

   • If possible, review the agent's code or settings to see how it processes FAQ data.
   • Check if there are any specific rules influencing the agent's behavior in this case.

3. Test with Controlled Examples:

   • Create some test questions that shouldn't match anything in the FAQ data store.
   • Observe how the agent responds to see if the issue is isolated.

4. Consider Adjustments:

   • If this behavior isn’t what’s intended, you may need to tweak the playbook, the bnr_faq tool’s settings, or the agent’s logic to ensure it strictly follows the intended search order (exact match from the FAQ first, then unstructured documents).

If the issue persists, you can contact Google Cloud Support for further assistance.

I hope it helps!

[cpetrenciuc]

Hi @dawnberdan ,

These are the settings for the FAQ data store.

I haven't changed anything to them since the creation of the FAQ data store.

And this is how the faq tool is configured in the agent.