Dialogflow CX - data store - how to filter data for a specific user

[eugene_d]

Hi, I have created a chat app in Agent Builder, and I have successfully integrated it with slack and a web front end, I've been able to set session variables for my authenticated users and I have used those parameters in my flows.

I want to query a data store for data that is specific to a user. I can query data if I just dump it into an unstructured data store, but its not specific to a user. From reading the docs I can see how data store filters work , so I created an unstructured (with meta data) store that uses structData and passes a uri to an html document with the user's private data. When I use this data store it seems that that unstructured document is never searched.

If I output "$request.knowledge.answers" it shows me the document I care about, but it didn't search inside it, it returns the metadata and a link to that document. Am I misunderstanding that the unstructured (with meta data) JSONL files are the correct way to create a data store and filter for private data or is their another mechanism to filter using a unstructured store? Even if I have no filter I am still just getting information about my unstructured document, it doesn't seem to be searched.

Some of the data I want to search for is actually structured, should I just be fetching this with a webhook and then using a generator on the results instead of trying to use a data store? The data store docs have only very basic examples.

[xavidop]

Hi!

For something such specific, what I would suggest is to call a webhook that calls the data store that contains the specific information of that user. This is the concept of the Knowledge Graph but moved to the Gen AI space. The webhook will contain the logic to call the specific data store.

I do not recommend to have all users information in the same data store.

You can also play with Grounding levels in the Gen AI Agent settings.

Best,

Xavi

[eugene_d]

Do you recommend a data store per user, for us that could mean many 10s of thousands of stores then?

[xavidop]

In that case, can you please explain to me what you are trying to achieve? Probably a data store does not fit your needs and you will need something more custom and advanced like a QDrant database with user metadata (userID for exmaple) in every single document to achieve what you are trying to achieve. and the retrieve that info using Langchain

[eugene_d]

For each of our users we perform a DNA analysis, tracking of lab tests and other information, I am trying to allow a chatbot to pull select pieces of information out of a large set of information for each user as they require it. Examples could be:

* which lab markers from my last set of tests are out of range
* what are the top 10 genetic risks identified for me
* what genetic variants do I have that put me at risk for heart disease

These all need data retrieval then generation, that's what is suggested that vertex AI can do with data stores. Retrieving data from an API (database) doesn't make sense for the application, you want indexed data that can be found and passed to a generative step to make sense of it

[xavidop]

yeah, so right now you cannot add metadata to documents on vertex AI data stores so it could be quite tedious and when you are performing the RAG you could get info from different patients BUT what you can do is the following:

1. when you are creating a document in a data store you can specify a name to that document, for example: user_XXXXX.pdf
2. then, when you are calling the datastore from Dialogflow, you can specify a filter (give me all the docs that contains user_XXXXX) to provide better information: https://cloud.google.com/dialogflow/vertex/docs/concept/data-store-agent#filter

I think that will solve your issue with the unstructured data.

Let me know what you think,

Best,

Xavi

[bakiasib]

What if you don't want to use Dialogflow, you just want to use the search service for vertex. Is there no way to specify a filter such as give me all the docs that contains user_XXXXX? Similar to the siteSearch filter?