Transferring Data from Amazon S3 Glacier to a Storage Bucket?

[muneebAr]

I am trying to transfer a large volume of data (250 TB) from Amazon S3 Glacier to a storage bucket. Still, it appears that the Storage Transfer Service does not support transfers directly from S3 Glacier to a storage bucket. Instead, it seems that the data must first be restored to an S3 bucket before it can be transferred to the desired storage bucket.

This process will incur charges for data retrieval from S3 Glacier to the S3 bucket, which could be substantial due to the large volume of data & restore speed

Is there any way to transfer data from S3 Glacier to a storage bucket without first restoring it to an S3 bucket? Does it matter if the data was stored in S3 Glacier directly or if it was stored using a life cycle rule?

Any insight or suggestions would be greatly appreciated.

[robertcarlos]

Hi @muneebAr 

As of the moment Amazon S3 is currently supported by Google Cloud Storage Transfer Service based on this documentation on creating transfer for Amazon S3.

As a workaround, you may use this Apache Airflow link on GlacierToGCSOperator from Cloud Composer however it is not a guarantee that it would fully work as per editor's note:

Please be warn that GlacierToGCSOperator may depends on memory usage. Transferring big files may not working well.

You may also file a feature request. We don't have a specific ETA for this however you can track its progress once the ticket is created and submitted to the proper team that will handle such requests.

[muneebAr]

I don't think it supports "Glacier Deep Archive" "Glacier Flexible Retrieval" Object Class in S3 bucket.

I get the following error "InvalidObjectState: The operation is not valid for the object's storage class" when I attempted to transfer through the Cloud Storage Transfer Service. Although "Glacier Instant Retrieval" is also the archive data type but it was transferred successfully so it appears that only 2 types are not supported