Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials.

[trushasahu]

I am trying to read data from the external bigquery table (i.e. linked to google sheet) in apache beam pipeline as below and later loading into bigquery table only. But the code getting failed with below error during reading data,

Error: "Access Denied: BigQuery BigQuery: Permission denied while getting Drive credentials."

with beam.Pipeline(options=google_cloud_options) as p:
            # Step 1: Get URL list from BigQuery table
            urls = (
                p | 'Get URLs from BQ Table' >> beam.io.ReadFromBigQuery(
                    query="""
                        SELECT cast(id as INT64) AS site_id, url
                        FROM `my-project.my_dashboard.MAP_URLS_GSHEET`
                    """,
                    use_standard_sql=True
                )
            )

I am using service account which have access to the google sheet as viewer and also to BQ database.

Note: The same code is working in Dataflow while using native BQ table instead of external table.

Could you please help here , how to solve this issue i.e. specially when running through dataflow?