Best Practices for Handling Large CSV Files in GCP using Cloud Function

[Samvardhan]

Hi there,

I have a CSV file of around 10GB stored in a GCP bucket. I perform basic operations on this file, such as:

• Adding a row_count column
• Renaming columns
• Formatting the CSV into a proper structure

These are relatively lightweight formatting tasks and not intensive data processing.

Currently, I am using a Cloud Function with the following configuration:

• Memory: 8GB
• CPU: 2

Is this setup sufficient for such tasks? Or should I consider using Dataflow or another service?
What would you recommend as a best practice for handling such operations?

Looking forward to your suggestions!