Vertex AI Agent Builder Media Recommendation Return Deleted Data

[chalermdej_l]

I have trained a recommendation model using the Vertex AI Agent Builder.

The model trains with enough user events and documents and it finishes training and is ready to query

I try and send the recommended request to my model using the endpoint in this document

https://cloud.google.com/generative-ai-app-builder/docs/reference/rest/v1beta/projects.locations.dat...

validateOnly has been set to False somehow the model return the document that is already deleted  from the data stores 

example this doc id 46314 return in the recommend result

But this document is already deleted from the data store sending a delete request from this endpoint

https://cloud.google.com/generative-ai-app-builder/docs/reference/rest/v1/projects.locations.dataSto...

Result in error not found below

 

Sending a get request to this document also doesn't work as it already deletes

Based on the recommended document in the vertex agent builder the response will never return EXPIRED or DELETED document

So why is the model still return this document even through it has been deleted?

I have try to run the tunning operation but doesn't seem to help 

The app is train with Recommendations type and media content : Recommend for you type, CLick-through-rate (CTR) 

Anyone know why this happen? And how to fix this