Unable to handle bignumeric in spark

[nids_k]

I am facing below issue in spark code /

We are running spark code using dataproc serverless batch in google cloud platform. Spark code is causing issue while writing the data to bigquery table. In bigquery table , few of the columns have datatype as bignumeric and spark code is changing the datatype from bignumeric to numeric while writing the data. We need datatype to be kept as bignumeric only as we need data of 38,20 precision

Could we cast a column to bignumeric in spark sql dataframe like below code for decimal:

we have used below code while loading the data to bignumeric column in bigquery

df= spark.sql("""SELECT cast(col1 as decimal(38,20)) as col1 from table1""")

[Roderick]

Hi @nids_k,

I hope I understand your question! 

Yes, you can cast a column to bignumeric in Spark SQL DataFrame using the cast function. Here is an example:

df = spark.sql("""
SELECT cast(col1 as bignumeric(38,20)) as col1
from table1
""")

This code will create a DataFrame with a column called col1 that is of type bignumeric with 38 digits of precision.

Please note that the cast function can only be used to cast a column to a type that is supported by the target data source. In this case, the target data source is BigQuery, and bignumeric is a supported data type.

Let me know how it works out!

[smittra]

How to write data from PySpark into BigQuery BigNumeric datatype?

I have defined a column as DecimalType(38,18) in PySpark. But it is only compatible with Numeric datatype of BigQuery. What datatype in PySpark needs to be used to write to a BigNumeric column in BigQuery ?