This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
I'm trying to set up a Dataproc workflow where I extract data from one
API and load it into BigQuery, but in order to do this, I need to get a
secret api key from Secret Manager, and herein lies the problem.I get
the following error message, regardle...
If by chance my earlier solution didn't help, these steps will also
work:Save the following file into a bucket:
https://github.com/GoogleCloudDataproc/initialization-actions/blob/master/python/pip-install.shIf
using the GUI:When creating a new cluste...
Thanks for your thorough feedback @ms4446 ! That makes a lot of sense. I
really liked your first suggestion of Installing the Library in the
Conda Environment, so I spent a lot more time looking into how to make
this happen. After trying various opti...
Got it - thanks for the clarification! (And thank you for continuing to
respond to me!) In regards to your reply - PySpark Environment: Ensure
that the PySpark environment is using the same Python interpreter where
the google-cloud-secret-manager lib...
Thank you for getting back to me so quickly! In response to your
suggestions:Initialization Bash Script:Verify Installation Command:
Ensure the pip3 install --user google-cloud-secret-manager command is
correctly executed in the initialization bash s...
