gRPC PERMISSION_DENIED Error & Streaming Hangs in Databricks on Google Cloud

[minhhung10l]

I'm running multiple streaming jobs on Databricks in the Google Cloud environment and am encountering issues that appear to be related to gRPC session management.

I implemented a custom streaming query listener to capture job status events and upsert them into a Delta table. This listener works fine when processing a single streaming table (upsert to a Delta table in a GCS bucket), but when I run approximately 10 or more streaming tables concurrently, I receive this error:

warnings.warn(f"Listener {str(listener)} threw an exception\n{e}")
/databricks/spark/python/pyspark/sql/connect/streaming/query.py:561: UserWarning: Listener <...CustomStreamingQueryListener object...> threw an exception
<_InactiveRpcError of RPC that terminated with:
    status = StatusCode.PERMISSION_DENIED
    details = "Local RPC without associated session."
    debug_error_string = "UNKNOWN:Error received from peer ..." >

This suggests that there might be an issue with gRPC—specifically, it seems related to session management or permissions—when handling multiple streaming queries on Google Cloud.

Questions:

1. Resolving the gRPC Error:
   How can I address the "Local RPC without associated session" (PERMISSION_DENIED) error when performing Delta upserts in the onQueryIdle event? Could this be due to gRPC session management issues between Databricks and Google Cloud, and are there configuration tweaks or best practices to mitigate this error in a high-concurrency environment?

Note: If you'd like to see the full code context, please refer to my detailed post on the Databricks Community forum here.