This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Reducing Dataproc Serverless CPU quota

Posted on 11-30-2022 02:54 AM
rudraSingh117
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hey guys! Can I get some clarity here? I am having problems with running spark jobs on Dataproc serverless. 

Problem: The minimum CPU memory requirement is 12 GB for a cluster. That doesn't fit into the region CPU quota we have and requires us to expand it. 12 GB is overkill for us; we don't want to expand the quota.

Details: This link mentions the minimum requirements for Dataproc serverless: https://cloud.google.com/dataproc-serverless/docs/concepts/properties

They are as follows: (a) 2 executor nodes (b) 4 cores per node (c) 4096 Mb CPU memory per node(memory+ memory overhead)

Hence, a total 12 GB of compute memory is required. Can we bypass this and run Dataproc serverless with less compute memory?

0 1 542
Topic Labels
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
RC1
Bronze 4
Bronze 4
Reply posted on --/--/---- --:-- AM

@rudraSingh117 

If you have some idea about what data you will be processing than you check out dataproc clusters and select the cluster as per your choice. Schedule using workflow in dataproc , which will create a cluster , run your job , delete your cluster. 


0 Likes