How many time take Datastore to refresh if Im currently doing changes on my website?

[Mizar]

hi Team, I have been adding more articles for my website, somehow sometimes the bot is not responding to lastest articles added, I created the data store based on my website 7 months ago and since that I have been adding articles each week.

I looked at documentation to see if I need to refresh or recreate the data store but didnt find anything.
could you please guide me, if I need to create a new data store, or if its refreshed each day or if I can change this refresh schedule.

[dawnberdan]

Hi @Mizar,

Welcome to Google Cloud Community!

Your data store, created 7 months ago, likely has not  been automatically updated to include your new articles. This is because Vertex AI Agent Builder doesn't automatically refresh your entire website's content on a regular schedule. Here are potential solutions

1. Manual Refresh (Recrawl): Vertex AI Agent Builder provides a way to manually refresh specific web pages. You can use the recrawlUris method to tell the system to crawl your new articles. You'll need to provide the URLs of the new articles you want to include.
2. Automatic Refresh: If you're using Advanced Website Indexing, you can set up a schedule for automatic refreshes. This ensures your data store stays up-to-date. This documentation will explain how to configure this feature. You might need to use Google Cloud Scheduler or a similar tool to automate the process.
3. Recreating the Data Store (Rarely Necessary): Recreating the data store is a drastic measure and should only be used if other methods fail. It involves starting from scratch and can be time-consuming.

Additional Tips:

• Start by manually refreshing your new articles using the recrawlUris method. This is the quickest way to get your bot responding to them.
• If you're adding articles often, schedule regular refreshes to keep your data up to date.
• Check your bot's performance after refreshing. Investigate if any issues arise.

I hope the above information is helpful.

[malam]

Hi all, 

I am facing a similar issue. My website datastore (advanced indexing on) hasn't had any refresh in two weeks and my manual refresh (recrawlUris curl command) shows successful indexing but the uris are still unavailable in the datastore. 

I've also noticed a drop in the datastore size over time without any monitor of this in the activity panel of the datastore. 

I posted about part of my issue: https://www.googlecloudcommunity.com/gc/AI-ML/Dialogflow-CX-Website-Datastore-Unpredictable-Behavior...

Any help would be greatly appreciated