Vertex AI Search: Get all the results

[mmargowski]

I am using the Vertex AI search in my current project.
When I search for a term I get an expected result of 5k but when I use the API to retrieve all 5k results I only get around 30 pages with 10 items each. I never get more than 300 results per search even though google is showing a lot more expected results.

Is there a way to get all the available results?

[shanefeely]

Hey there, I was able to connect with one of our internal teams and they shared the following answer with me:

- The number of results is controlled by PageSize in here: https://cloud.google.com/generative-ai-app-builder/docs/preview-search-results#genappbuilder_search-...

- The Maximum is 100 as shown here: https://cloud.google.com/generative-ai-app-builder/docs/reference/rest/v1/projects.locations.collect...

- You may be able to use "offset" in the second link to retrieve more than 100, in multiple calls. i.e. retrieve 1-100 in the first call, 101-200 in the second call and so on

Hope this helps!

[mmargowski]

Thank you for the reply.

I tried it and ran into the same limitations with the offset and the page size:
1. When adding a pageSize of 100, I still get a maximum of 25 results returned per page. 
2. Even with using the offset I can rarely retrieve more than 3-400 results.
For example: I tried a search and go this in the response:

"totalSize": 65600
I then sent a query with an offset of 400 and got 0 results returned.

This is my query:

{
"query": "TEST",
"pageSize": 100,
"offset": 400,
"queryExpansionSpec": {
"condition": "AUTO"
},
"spellCorrectionSpec": {
"mode": "AUTO"
},
"contentSearchSpec": {
"snippetSpec": {
"returnSnippet": true
}
},
"params": {
"user_country_code": "DE"
}
}

It basically stops at the same number of results as I could retrieve using the pageToken from the previous page.