About iost

iost · 10-16-2023

What is the reccomended way to access secrets from secret manager in a dataflow (python) pipeline ?

iost · 10-02-2023

I am trying to run a pipeline tha wirtes kafka messages to parquet file. But i can't and it outputs the error 'ValueError: GroupByKey cannot be applied to an unbounded PCollection with global windowing and a default trigger'. But non global window is...

iost · 10-17-2023

@ms4446 i didn't ask about database connections, but accessing secrets from secret manager. Could you help?Thank you

iost · 10-10-2023

as i have told you, i have tried windowed writes and custom triggers and due to the problem that is decribed in the github issue in my previous message, i get the same error.My question is how it worked for you when custom writing isn't supported in ...

iost · 10-09-2023

hello, according to this https://github.com/apache/beam/issues/25598, Only batch workflows support custom sinksI am wondering how is it possible to work for you, It still doesn't work for me

iost · 10-06-2023

I am running code from dataflow notebooks and i press a button on UI (run all cells) and then a job is created on dataflow. Beam 2.50,pyarrow 13, Python 3 Can you please give info about your enviroment you run the code, versions and the full pipeline...

iost · 10-05-2023

I have tested your solution and still get the same error!Can you please test your solution on dataflow to check that is working?

My Stats

PauloAll's Bio

Badges iost Earned

Recent Activity

Secrets and Dataflow python

kafka to parquet

Re: Secrets and Dataflow python

Re: kafka to parquet

Re: kafka to parquet

Re: kafka to parquet

Re: kafka to parquet