Preserve timestamp while migrating data from S3 to GCS

[rohankalhans]

Hi,

We are migrating more than 500TB of data from AWS S3 to GCS and due to compliance and audit, we always need the original timestamp to use by audit teams to create reports. I checked several GC docs, is there a way to preserve the original (or source) timestamp (of AWS) after migrating data to GCS?

Thank you! 

[robertcarlos]

Hi @rohankalhans,

Welcome to Google Cloud Community!

Use of AWS CloudWatch metrics in collecting metrics from AWS accounts was deprecated as of August 21, 2023 and only existing AWS Connector projects will be supported up until August 21, 2024. 

This is being replaced by Prometheus CloudWatch exporter which is an open source Prometheus exporter that can collect AWS CloudWatch metrics from the CloudWatch API for AWS services and send them to Cloud Monitoring.

You may also use BindPlane which can collect their own data from their on-prem and hybrid cloud platforms and send it to Cloud Logging for analysis. You may check this documentation on observIQ and BindPlane for more information.

Please check the documentations/references below that you may find useful:

• Cloud Logging for Storage Transfer Service 
• Monitoring and logging resources 
• Cloud Logging for Storage Transfer Service 
• Monitor transfer jobs 
• File system transfer logs 
• Structured logging 
• How can I get AWS cloudwatch logs into GCP stack driver logs viewer? 

Hope this helps.

[rohankalhans]

Hi @robertcarlos 
Thanks for the answer however, can you please elaborate?
Based on your answer, I checked the aforementioned tools but still, it wouldn't able to capture the source timestamp.

[rohankalhans · Accepted solution]

Hey @robertcarlos 
We found the solution. I have posted the solution here.