Why Does The Google Search API Disallow More Than 100 Results? How Can I Get More?

[ronnieroyston]

I'm using the Google Client Libraries to call the Custom Search API. My specific query returns 300,000 results. I want to iterate them all and perform custom filtering with a script. However, the API refuses to return more than 100 links / results. 

Is there a way to get more?

https://developers.google.com/custom-search/v1/reference/rest/v1/cse/list

start

integer (uint32 format) The index of the first result to return. The default number of results per page is 10, so &start=11 would start at the top of the second page of results. Note: The JSON API will never return more than 100 results, even if more than 100 documents match the query, so setting the sum of start + num to a number greater than 100 will produce an error. Also note that the maximum value for num is 10.

[christianpaula]

Hi @ronnieroyston,

I would highly suggest contacting Google Cloud Search Support about this concern.

Thank you

[ronnieroyston]

Cloud Search "searches across your organization's content in Google Workspace services or third-party data sources." I don't think it is the same as general web search. In fact, there is no general web search API. There is only "custom search" API. However, you can run general web search using that API.

[Ali-Alani]

Hi Google Cloud Community,

I hope this message finds you well. I'm currently working on a Ph.D. project and using the Google Client Libraries to interact with the Custom Search API. However, I've encountered a challenge where the API seems to limit the number of results to 100.

My project involves iterating through a substantial number of results and applying custom filtering. If anyone has faced a similar situation or has insights on how to overcome this limitation, I would greatly appreciate your guidance.

Thank you in advance for your assistance!

Best regards,

[ronnieroyston]

Bing might work? https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/quickstarts/rest/nodejs