Vertex AI Search - can it index non-public Google sites behind authentication?

[RVinegas]

In our testing, Vertex AI Search can only search public sites. Is there a way for Vertex AI Search to be able to index Google sites behind authentication?

[Poala_Tenorio]

Vertex AI Search primarily focuses on indexing public content across the web. This means that it's optimized to search and retrieve information from publicly available websites. However, when it comes to indexing content behind authentication, such as Google Sites requiring login credentials, Vertex AI Search might not directly support this feature.

To index content behind authentication, you might need to explore other solutions or potentially create a workaround by making the content accessible without authentication, at least for the crawler or indexing mechanism.

Sometimes, for indexing purposes, creating a parallel public-facing version of the content might be necessary. This version would be accessible to the Vertex AI Search crawler without authentication. Also, implementing a proxy or middleware that handles the authentication and presents the content in a way that's accessible to the search engine crawler could be an option. However, this approach would require careful consideration of security and privacy aspects.

[RVinegas]

What are the recommended best practices or steps for creating a parallel public-facing version of the content that is accessible to Vertex AI Search? Are there any specific tools or strategies you would suggest for implementing this approach effectively while ensuring security and privacy considerations are addressed?