How to connect Elasticsearch to Jupyter Notebook on GCP

[shivang1234]

Hi,

I am working on an ML project which requires to use Transformers and Elasticsearch.

For Transformers, I have created a Jupyter Notebook instance on GCP.

For Elasticsearch I will create another instance on CGP.

As part of project requirement, I need to access Elasticsearch in Jupyter Notebook through port 9200. I need to ingest data in Elasticsearch and run search queries which can fetch relevant information from Elasticsearch DB and give it in Notebook.

My question is that if I create two separate instances of Notebook and Elasticsearch and if i try to connect Elasticsearch through Jupyter Notebook via port 9200, will i be able to connect and perform the above mentioned operations ?

If NO, then what is the procedure to do so ?

[ErnestoC]

Can you elaborate on how you have set up the Elasticsearch and Jupyter notebook instances? On which compute service is your Elasticsearch instance running?

[shivang1234]

Hi,

Below is the Screenshots of the way i have setup Jupyter Notebook instance and Elasticsearch instance.

1.

Vertex AI Workbench - Jupyter Notebook

This is a link of the screenshot of the Vertex AI workbench where I have created a Jupyter Notebook instance.

2. Below is the screenshots of the way I have accessed Elasticsearch through marketplace. Once I click on the service provider, the second screenshot is the landing page of Elasticsearch Cloud where also, I have signed and created the account.

Elasticsearch in Market PlaceLanding Page of Elasticsearch once we click on provider

 3. I want to access the Elasticsearch through "http" request made through my Jupyter Notebook and the screenshot of the codes which I want to run in my Jupyter Notebook is attached in below links.

Code Snippet 3

Code Snippet 2

Code Snippet 1

 Please Note that I want to access Elasticsearch version 8 onwards, as version 8 onwards, there is a facility to ingest "Dense Vector" in the database.

[ErnestoC]

Using the Elasticsearch managed service from the GCP marketplace, you would need to use the [provided guide to connect using a Python environment (in this case being the Jupyter notebook). Additional methods of connecting to the JSON API of the service are also included in the documentation.

If you would like to exclusively connect directly to the deployment using port 9200, creating a deployment within a VM instance in GCE should be more appropriate. This is due to the fact that a Vertex Workbench Notebook is also hosted on a GCE instance, and can be set up to be in the same VPC as a VM hosting Elasticsearch.