Vertex AI Search & Conversation app Automatic training Issue

[kbujari]

Hi. I am using GCP Vertex AI Search & Conversation app for my project. I have created a Cloud Storage and uploaded my data "Help_Data". I used that dataset in my project in Vertex AI Search & Conversation the name is "SearchBot". Now as everything works fine I want to automate the process of re training the model. Each week once I am getting new data and through my Python code I am uploading those data to Cloud Storage but I don't know how can I automatically transfer those data to my data store in Vertex AI Search & Conversation "SearchBot" app and re-train the mode. I have tried with schedulers I couldn't manage to do it. I also tried through python code to transfer data from Cloud Storage to Data Store in Vertex AI Search & Conversation. Can you help me to automate this process? How I can train my model automatically each week with new data? Thank you!

[xavidop]

Hi @kbujari you will need to use the discoveryengine API and SDKS. Here you have an example in python: https://cloud.google.com/generative-ai-app-builder/docs/samples/genappbuilder-import-documents?hl=en

Best,

Xavi

[kbujari]

Hi Xavi,

Thank you for the reply. I am using this code but I get the error below (I hided my project id for the post):
google.api_core.exceptions.InvalidArgument: 400 Resource name "projects/project_id/locations/global/dataStores/test-data" cannot be parsed. Resource name prefix must be of the format "projects/<Your_Project_Number_or_Project_ID>/locations/*/collections/*/dataStores/*/branches/*"
 
See my code below:

credentials = service_account.Credentials.from_service_account_file('genai.json')

def import_documents_sample(

project_id: str,

location: str,

data_store_id: str,

gcs_uri: Optional[str] = None,

bigquery_dataset: Optional[str] = None,

bigquery_table: Optional[str] = None,

) -> str:

client_options = (

ClientOptions(api_endpoint=f"{location}-discoveryengine.googleapis.com")

)

# Create a client with authentication credentials

client = discoveryengine.DocumentServiceClient(credentials=credentials, client_options=client_options)

parent = f"projects/{project_id}/locations/global/collections/default_collection/dataStores/{data_store_id}"

request = discoveryengine.ImportDocumentsRequest(

parent=parent,

gcs_source=discoveryengine.GcsSource(

input_uris=[gcs_uri], data_schema="custom"

),

# Options: `FULL`, `INCREMENTAL`

reconciliation_mode=discoveryengine.ImportDocumentsRequest.ReconciliationMode.INCREMENTAL,

)

# Make the request

operation = client.import_documents(request=request)

print(f"Waiting for operation to complete: {operation.operation.name}")

response = operation.result()

# Once the operation is complete,

# get information from operation metadata

metadata = discoveryengine.ImportDocumentsMetadata(operation.metadata)

# Handle the response

print(response)

print(metadata)

return operation.operation.name

import_documents_sample(

project_id,

location,

data_store_id,

gcs_uri

)

Thank you for your help!

[xavidop · Accepted solution]

uh, that took me a while as well. the format should be something like this:

projects/{project}/locations/{location}/collections/{collection_id}/dataStores/{data_store_id}

[kbujari]

Hi Xavidop,

Thank you very much for your reply! However I have one more question, what do you mean with {project} is it project name or id, and I don't have collection, should I keep /collections/{collection_id} this in link or remove it?

Thank you in advance!

[xavidop]

it always refers to project ID not project name for collection should be /collections/default_collection

[kbujari]

Thank you very much for your help. It works now! 

[xavidop]

AWESOME!

[sohilagrawal]

I am also trying to do conversation with my dataset, but conversation is not happening on my created search app from my imported data, what could be the problem, is there a way to debug the problem?

[xavidop]

you have to create a conversation app not a search app