System instructions ignored after grounding Gemini in Vertex AI Data Store

[ElliotK-IB]

I'm trying to build a multi-turn chatbot that can answer customer questions by referencing docs in a Data Store. I also want to give it a specific persona and behavior.

What works:

• Gemini API + System Instructions
• Gemini API + Grounding in Vertex AI Data Store

What doesn't work:

• Gemini API + System Instructions + Grounding in Vertex AI Data Store

After the first 1-2 turns, Gemini continues to reference docs in the Data Store to answer questions but seems to "forget" its system instructions (loses the designated persona and behavior).

I've mainly been using this example in the generative-ai GitHub for reference: Getting Started with Grounding with Gemini in Vertex AI

Thanks in advance for your help.

[cassandramae]

Hi @ElliotK-IB,

Welcome to Google Cloud Community!

Sorry to hear that your system instructions are being ignored. System instructions are prompts that guide the model's overall behavior and persona while grounding focuses on providing information from your data store. It could be that the LLM may prioritize grounding information over the system instructions which might cause this behavior.

With this, you may consider the following solutions:

• By explicitly reintroduce the system instructions from previous turns
• Store the conversation history in a format that the LLM can easily process. You can then insert this history into the prompt for every new turn
• Use filters or queries to retrieve only the most relevant documents for each turn that can help the modele focus on the task at hand

Hope this helps.

[adasdasd]

• Gemini API + System Instructions + Grounding in Vertex AI Data Store