This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Good dayI have a datasteam stream ingesting tables into cloud storage
(4000+ tables at a 60 second file rotation interval.)So, as you can
imagine, I am generating a very large number of small gzipped json
files. Millions per day.I want to perform dat...
Good dayI have a datasteam stream ingesting tables into cloud storage
(4000+ tables at a 60 second file rotation interval.)So, as you can
imagine, I am generating a very large number of small gzipped json
files. Millions per day.I want to perform dat...
I am trying to create a datastream stream using the python client
library: stream = datastream_v1.Stream( display_name="TEST_STREAM",
source_config=datastream_v1.SourceConfig(
source_connection_profile="projects/{PROJECT_NUMBER}/locations/europe-west...
I have set up a large number of tables as biglake tables connected to
cloud storage uri's, and then I have set up a large number of
materialised views on top of the biglake tables.The problem I have seen
is that when the "last modified date" of the b...
Good day allI am in the process of backfilling and streaming a large
number of tables (>1000) using datastream.When I click start stream it
works for a few minutes and then I get the following error:""(1129,
"10.200.0.10' is blocked because of many c...
Once again thank you for the answer.I think from the error: The Apache
Avro library failed to read data with the following error: Field in
reader schema 'new_col' without default value not present in writer
schema, It is likely to be 6.Do you perhaps...
Thank you for the detailed response. I am, however, still struggling
with this issue.Should there be specific configurations made to
datastream, gcs or bigquery for this to work, or is it possible to set
up using just the console?I'll add a few more ...
