This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

How do you orchestrate and schedule Dataproc serverless batch jobs ?

Posted on 07-18-2022 07:40 AM
RC1
Bronze 4
Bronze 4
Reply posted on --/--/---- --:-- AM

I have multiple pyspark jobs with dependencies. I want to orchestrate and schedule it. How it is possible to do this in  Dataproc ( serverless ) batch jobs ?

0 1 1,472
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
josegutierrez
Staff
Reply posted on --/--/---- --:-- AM

Currently there is no way to schedule batch jobs using pyspark. There is a way to schedule jobs by creating a a Dataproc workflow template In Cloud Scheduler that runs a Spark PI job see this document. 

You can File a Feature Request at Issue Tracker.

0 Likes
Top Solution Authors
View all