Datastream Oracle Source Unsupported Data Type CLOB

[IceElm]

I'm trying to ingest an object from an Oracle source via Datastream, but one of the columns is not being pulled across. The schema information is saying that the source data type is CLOB and the Datastream documentation says this is supported (as a STRING) [1].

However, when I use BigQuery as the destination the column is ignored/ isn't even created. When I try and use GCS (JSON w/ Avro schema) the data for the column comes through as nulls and the schema information says:

{"name":"<COL_NAME>","type":{"type":"null","logicalType":"unsupported"},"default":null}

I'm expecting this data to be present and the source schema in the datastream UI even says 'NULLABLE: No'

Any insight into why this might be happening or how to rectify would be appreciated. Is it the case that the field could be too large at source for either BQ or GCS to handle?

[1]  https://cloud.google.com/datastream/docs/destination-bigquery#map-data-types

[mcbsalceda · Accepted solution]

Hi @IceElm,

Welcome to Google Cloud Community!

You might want to try enabling streaming of large objects for your Oracle source in Datastream. This setting allows Datastream to stream CLOB data in chunks instead of skipping or truncating it, which could be why the column is missing or showing up as null.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[IceElm]

Hi @mcbsalceda,

I've just configured this on my stream and it's now bringing the data as expected. I had no idea this was a flag I could use.

Thanks so much for the help