Cloud Search with Google Spanner Integration

[Roryon]

Hello all,

I have a use case where I have data stored in Spanner that I want to be able to quickly search from a REST API that will be on GCP Cloud Run.  I am trying to determine what the best choice would be for the search platform/index that sits between the REST API and Spanner.  Would Cloud Search fit this role?  If so, can anyone point me to some documentation on how to seed Cloud Search with Spanner data?  If this wouldn't fit my use case, can anyone recommend a better solution?

Thank you.

[ms4446]

Google Cloud Search is primarily designed for searching across G Suite products like Gmail, Drive, Calendar, Docs, and more. It's not specifically tailored for searching database content like that in Spanner.

For your use case, you might consider the following options:

• BigQuery: BigQuery is a fully managed, petabyte-scale analytics data warehouse that can be used to index and search Spanner data. BigQuery is a good option if you need to search large volumes of Spanner data in real time.
• Elasticsearch: Elasticsearch is a distributed search and analytics engine that can be used to index and search Spanner data.Elasticsearch is a good option if you need to search Spanner data in real time and require a high degree of customization.
• Druid: Druid is a distributed, column-oriented data store that is optimized for real-time analytics.Druid is a good option if you need to search Spanner data in real time and require high performance.

[Roryon]

After looking at those three options further, I think Elasticsearch will fit our needs the best.  How is Elasticsearch usually deployed on GCP and how do I index the Spanner tables on Elasticsearch and keep them in sync?  Any documentation you can point me to would be helpful.  Thank you!

[ms4446 · Accepted solution]

There are a few ways to deploy Elasticsearch on GCP. One popular option is to use Elastic Cloud, which is available on the Google Cloud Marketplace. Elastic Cloud is a managed Elasticsearch service provided by Elastic, the company behind Elasticsearch.

To deploy Elasticsearch on GCP using Elastic Cloud:

1. Create a Google Cloud Platform project.
2. Access the Google Cloud Marketplace and set up an Elastic Cloud deployment.
3. Configure and launch your Elasticsearch cluster.

Once you have deployed Elasticsearch on GCP, you can index your Spanner tables on Elasticsearch. While tools like Logstash can be used to ingest data into Elasticsearch, there isn't a direct plugin for Google Cloud Spanner. You might need custom integrations or intermediary steps to get data from Spanner into Elasticsearch.

To keep your Spanner tables in sync with Elasticsearch, you'd typically use tools like Logstash or develop custom integrations. The previously mentioned Elasticsearch River plugin has been deprecated and is not recommended for modern deployments.

Here are some additional resources that you may find helpful:

• Elastic Cloud on Google Cloud Marketplace
• Logstash documentation

[ms4446]

Update: 9/10/2024

Spanner recently introduced Full Text Search. This game-changing feature allows customers to perform robust, scalable searches directly within Spanner, eliminating the need to export data to external search platforms.

This advancement can streamline your architecture, reduce operational complexity, and enhance search performance for your applications.

Learn how to leverage this feature by exploring these resources:

• Building Applications with Spanner Full Text Search
• Spanner Full Text Search Documentation