Multi part upload api in Google cloud storage

[Chethan]

I have a requirement of supporting upload of large files (probably 100 / 200 gigabytes). I would like to know which api would be best for such scenario. I'm using node js in my app. There is composite parallel upload and there is XML api, Need help in choosing the kind of api that would be best and would be great if I can get some stats on time taken to upload considering 100 Mbps network

Thanks 

[christianpaula]

Hi @Chethan,

Welcome to Google Cloud Community!

For uploading large files to Google Cloud Storage, the recommended API to use is the resumable upload API. This API allows you to upload large files in chunks, and if the upload is interrupted, it can resume from where it left off. This is useful for uploading large files over a potentially unreliable network connection.

The composite parallel upload API is also an option, but it's mainly used for uploading many small files in parallel. It's not as well-suited for uploading a single large file.

The XML API is the older version of the API, it's not recommended to use it for new development.

Upload speed will depend on various factors such as the size of the file, the available bandwidth, and the number of parallel upload threads. In general, you can expect upload speeds of around 100Mbps for a 100Mbps network.

You can use the npm package '@google-cloud/storage' for Node.js to interact with Google Cloud Storage

Google Cloud Storage provides extensive documentation on the resumable upload API and the composite parallel upload API. Here are some links to get you started:

Resumable upload API documentation
Composite parallel upload API documentation
The npm package '@google-cloud/storage' also have a documentation page:
https://googleapis.dev/nodejs/storage/latest/index.html

You can also find examples and sample code for different scenarios in these documentations.

Thanks

[Chethan]

Thanks @christianpaula , I'll try this out.

When I mentioned XML API, I meant XML multi-part upload API. I was looking for a JSON API similar to AWS S3 multi-part API.

Also, In Resumable uploads, it is mentioned that we could provide chunk size. Would we call multiple APIs for each chunk? If so, Could you please tell me if we could call them parallelly or should it be in sequence? 

Thanks

[Chethan]

Hi @christianpaula , I have tried resumable upload option, I see that there are 2 options of uploading a file in one go as well as initiating a resumable upload and sending file in chunks using that url. I observe that sending in chunks sometimes result in exceeding update rate limit. Is there any restriction in number of parallel writes to an object ? How do I cancel in such error cases. Also, if a file is streamed in just one http request and if for some reason the server fails, Would it automatically upload from the stopped byte or would it be like a new full file upload when retried next time.

Thanks

Chethan