I understand your predicament. It seems like you've deleted all the batches but the disk space is still not freed up. Here are a few steps you can take:
-
Check for any active jobs or instances: Even though you've deleted all batches, there might be some active jobs or instances that are still running and consuming disk space. You can do this by navigating to the Cloud Console and checking the status of your jobs or instances.
-
Delete unused disks: If there are any unused disks, you can delete them to free up some space. You can do this by navigating to the 'Disks' section in the Cloud Console and deleting any disks that are not in use.
-
Check for snapshots: Sometimes, snapshots of your disks can also consume disk space. You can check for any snapshots and delete them if they are not needed.
As for the billing, you're right that increasing the quota might lead to additional costs. However, controlling the disk size using the spark.dataproc.driver.disk.size and spark.dataproc.executor.disk.size properties can help manage the costs. The minimum required might be 250GB total, but you can adjust these values based on your specific needs and budget.
LinkedIn
Twitter
