Issues with dataproc serverless connection to iceberg biglake tables

[yoris30]

Hi all, 

I've been trying to connect to an Iceberg table that I created through a bigquery stored procedure. My goal is to perform CRUD operations using pyspark via dataproc serverless. To achieve this, I've been working on creating an interactive template to run spark in jupyterlab workbench.

Despite my efforts, I'm facing significant challenges in establishing this connection. The process has been more complicated than anticipated, and it's causing me a lot of stress. Has anyone successfully navigated this setup or can offer guidance on the best practices to follow? Any help or insights would be greatly appreciated.

[ms4446]

Connecting to an Iceberg table in BigLake using PySpark on Dataproc Serverless is indeed a complex but it is achievable. This process involves several steps, starting with setting up your environment. First, ensure that your Dataproc Serverless cluster is properly configured, and install the Iceberg library in your PySpark environment to enable interaction with Iceberg tables. Additionally, your JupyterLab workbench on Dataproc should be set up with the necessary packages to handle BigLake and Iceberg.

Next, configuring the Iceberg table within BigLake is very important. Verify that the table is correctly set up and that you have the required permissions to perform CRUD operations. Testing connectivity by running a simple read operation using the Iceberg API can help ensure that your setup is functioning as expected.

Creating a Spark session in JupyterLab is the next crucial step. This session must be configured to connect to Iceberg in BigLake by specifying details such as the custom Spark catalog, the catalog type, and the storage location. Proper configuration of the Spark session is necessary for seamless integration with Iceberg features.

Once the setup is complete, you can perform CRUD operations on your Iceberg table. Reading data is straightforward using the spark.read.format("iceberg") method, while inserting data involves creating a DataFrame and appending it to the table. For updates and deletes, you can either use basic DataFrame operations or more advanced SQL-based approaches and Iceberg APIs, depending on the complexity of your requirements.

Troubleshooting is an integral part of the process. If you encounter issues, checking the logs of your Dataproc Serverless jobs can provide insights into potential errors. Starting with simpler tasks, like basic data reading, can help establish connectivity before moving on to more complex operations. Ensuring that your IAM permissions are correctly set up is also crucial for smooth operation.

Finally,  Implementing logging and monitoring through Logging can help track and resolve issues promptly. Data partitioning, based on frequently queried columns, can optimize performance, particularly when dealing with large datasets. Ensuring data consistency during multiple operations is vital, especially when handling concurrent updates and deletes.

[yoris30]

Thank you for the explanation. It seems there are no available docs with hands-on materials for this use case on Google, or I may have missed them. I would greatly appreciate it if you could provide any resources with hands-on examples or code for this case.

[sadeveloper]

If you were able to make progress on this, please let us know, I am trying to make sense of these as well.

How did you create the table using the proc, if you can elaborate?
Also, it looks like BigQuery has a new feature now called Iceberg Tables which can be mutated from BigQuery. This is in preview but seems very promising.

https://cloud.google.com/bigquery/docs/iceberg-tables