Using S3 interface with GCS, uploads are corrupted because server is storing embedded chunk-signatur

[JamesOwen]

I'm using the AWS S3 libraries, configured to connect to Google S3 interface (a simple migration scenario) and I can connect, browse buckets and folders, read content correctly etc. What fails is that when I upload a file to it, GCS does not strip the embedded chunk-signatures from the data. Before I rewrite the client code to bypass the S3 library and write http requests manually, can anyone tell me how to prevent this happening? To be clear, the AWS SDK uses x-amz-content-sha256 in the header with a value of STREAMING-AWS4-HMAC-SHA256-PAYLOAD (rather than including the hash in the header) which adds the hash to the payload. The Google server writes the hash codes into the file content, corrupting it.

[KirTitievsky · Accepted solution]

James, thanks for reporting this. This is a known issue.  At the moment, I can't offer a timeline on a resolution, but someone is working on this actively.  Your report makes it easier to escalate the importance of making this work properly.  In the meantime, I suspect if you need a solution within the next few months, writing it yourself is likely the most reliable way to success.  Sorry this is not better. 

[JamesOwen]

Thank you for replying. We'll construct a workaround which will probably end up being the preferred solution!

[nprime496]

Hi, I'm constructing scenarios for migration, I'd like to know if the problem is now solved.

If not, do you please have ressources for a custom solution.

[mahesh-gupta]

Hi @KirTitievsky 

we are working on s3 to GCS copy scenario. Please let us know if this problem is now resolved. 

[KirTitievsky]

Thanks for following up, @mahesh-gupta .  I haven't worked with the GCS team in some time so don't have new information. I wonder if anyone else watching is closer to this issue.

[JamesOwen]

Hi @KirTitievsky , @mahesh-gupta , I've re-tested using our original code that depends on the AWS S3 library to upload large files to GCS, and the resulting upload is no longer corrupted. Many thanks to the GCS team for fixing this, much appreciated.

Paul